Open source and self-hosted
Langfuse's core, including LLM-as-a-judge evaluators, annotation queues, prompt experiments and the playground, is MIT-licensed and runs in your own environment.
SERVICES / OPEN SOURCE
Self-hosted agent evaluations with open-source Langfuse.
Independent design and implementation of AI agent evaluations with open-source Langfuse: datasets and experiments, LLM-as-a-judge and code evaluators, human annotation and CI checks, running in your own environment.
Where open-source Langfuse fits
Recorded runs as evidence
Each agent run is recorded step by step, including model calls, retrievals and tool calls, so every score can be traced back to what the agent actually did.
Works with your framework
Integrations cover the OpenAI SDK and Agents SDK, LangChain and LangGraph, LlamaIndex, Pydantic AI and other OpenTelemetry-instrumented frameworks.
Testing agents with Langfuse
Datasets and experiments
Reviewed datasets with expected outputs run against each agent version as experiments, from the UI, the SDK or a CI pipeline, so changes are compared on the same cases.
Judges and code evaluators
LLM-as-a-judge evaluators score answers against your rubrics, and code evaluators check exact rules such as required fields, valid tool arguments and policy limits.
Multi-turn sessions
Conversations are grouped into sessions, so we can evaluate whether context, commitments and handoffs hold up across turns, not just single replies.
Common pitfalls we fix
Judges that see too little
A judge scores only the step it is attached to. We design evaluation targets and summaries so judges see the full context a decision needs.
Datasets that only mirror production
Sampled runs over-represent common paths. We add the edge cases, failures and unanswerable requests that matter for your release decision.
Paths no one checks
Langfuse records tool calls but leaves path checks to you. We build evaluators that compare actual tool-call sequences with acceptable reference paths.
How we help
Set up for evaluation
We help you self-host or configure Langfuse, instrument the agent components that matter, and organize projects, datasets and scores for evaluation work.
Calibrate the judges
Your reviewers label examples in annotation queues, and we tune judge prompts until their scores agree with expert judgment, then document the agreement.
Release checks in CI
Experiments run automatically on pull requests or before release through Langfuse's experiment tooling, with critical scores failing the build.
Make it routine
Version comparisons
Prompt, model and tool changes run against the same dataset version, with quality, latency and cost compared side by side.
Feedback into the test set
Annotated failures and user feedback are reviewed and promoted into datasets, so the regression set grows with real issues.
Team enablement
We train your team to maintain datasets, evaluators and experiments as the agent and Langfuse evolve.
From scope to handover
Start with your inputs
Representative tasks, expected outcomes, approved run samples, Langfuse access or a hosting environment, agent access and a domain reviewer.
Keep reusable assets
Versioned datasets, evaluator definitions, judge calibration results, a baseline experiment, prioritized findings and a documented rerun process.
Support after handover
We can retest fixes, extend coverage to new Langfuse workflows or move to recurring Managed Evaluations as your agent changes. Platform usage and licenses stay in your account.
Make your next decision with evidence.
Start with one workflow, clear success criteria, and an evaluation plan your team can use.