Datasets and experiments
Datasets of examples with inputs and reference outputs, versioned as they change. Each experiment runs one version of your agent over a dataset, and versions are compared side by side.
SERVICES / EVALUATION PLATFORMS
Agent evaluations in LangSmith that support real decisions.
Independent planning and implementation for teams using LangSmith to evaluate AI agents: reviewed datasets, trajectory and final-response evaluators, judges aligned with your experts and experiments that compare every change.
Where LangSmith fits
Evaluators of every kind
Code evaluators, LLM-as-a-judge evaluators, human review and pairwise comparisons sit in one place, alongside the recorded runs they score.
Not only for LangChain apps
LangSmith works with agents built on LangChain and LangGraph and with other frameworks and model providers, through its SDKs.
Testing agents with LangSmith
Final response
Did the agent deliver the right answer or outcome for the task? Scored against reference outputs and your business rules.
Single steps
Did the agent make the right decision at a key point, such as routing a request or choosing a tool, with valid arguments?
Trajectories
Did the agent take an acceptable path? Tool-call sequences are matched against reference paths, strictly or allowing valid alternatives, or reviewed by a calibrated judge.
Common pitfalls we fix
Datasets that only mirror production
Datasets built from recorded runs repeat what already happens. We add the edge cases, failures and unanswerable requests that matter for your release decision.
Judges nobody has checked
LLM-as-a-judge scores drift from expert judgment unless they are aligned. Unchecked judges can pass answers your reviewers would fail.
Paths that are too strict
Exact trajectory matching can fail an agent for taking a valid alternative route. We choose matching rules that reflect what your workflow actually allows.
How we help
Build reviewed datasets
Representative, difficult and failure-prone examples with reference outputs and expected tool paths, organized into splits and version-tagged for fair comparisons.
Align judges with your experts
Your reviewers label examples in annotation queues, and we use LangSmith's Align Evals workflow to tune judge prompts until they agree with those labels.
Test multi-turn conversations
Simulated users run realistic conversations across several turns, scored with evaluators we design for your workflow, with repetitions to measure variability.
Make it routine
Experiments for every change
Prompt, model and tool changes run against the same dataset version, with quality, latency and cost compared before release.
Release checks in CI
Evaluations written as tests with LangSmith's pytest or Vitest and Jest integrations run on pull requests, with critical checks failing the build.
Team enablement
We train your team to maintain datasets, evaluators and experiments, and to keep judges aligned as the agent changes.
Related services
Evaluation Assessment
A baseline for one LangSmith agent or workflow, with prioritized findings and a release recommendation.
Explore serviceHallucination Testing
Measure how often your LangSmith agent invents facts, policies or actions, and which fixes reduce it.
Explore serviceManaged Evaluations
Recurring tests as your LangSmith agent, prompts, models and data change, with a monthly quality report.
Explore serviceFrom scope to handover
Start with your inputs
Representative tasks, expected outcomes, approved run samples, LangSmith workspace access, agent access and a domain reviewer.
Keep reusable assets
Versioned datasets, evaluator definitions, judge alignment results, a baseline experiment, prioritized findings and a documented rerun process.
Support after handover
We can retest fixes, extend coverage to new LangSmith workflows or move to recurring Managed Evaluations as your agent changes. Platform usage and licenses stay in your account.
Make your next decision with evidence.
Start with one workflow, clear success criteria, and an evaluation plan your team can use.