Many steps, many failure points
A single request may involve intent recognition, retrieval, several tool calls and a final reply. An error anywhere can spoil the outcome even when the reply reads well.
OUR METHODOLOGY
Use the right judge for the right question.
Reliable agent evaluation combines objective checks, calibrated AI judgment and expert review. We choose methods around the evidence your workflow and decision require.
Why agents need a different approach
Testing a chatbot answer is not the same as testing an agent that plans, calls tools and changes records across many steps.
Non-deterministic by design
The same input can produce different paths and results. Evaluation must measure distributions, not single runs.
Uneven capability
Harvard Business School research describes AI capability as a "jagged frontier": strong on some tasks, unexpectedly weak on similar-looking ones. Only testing on your tasks shows where your agent's edge lies.
Four complementary judges
Code-based checks
Best for objectively verifiable conditions: valid JSON, required fields, tool arguments, state changes, arithmetic, executable code and business-rule assertions. Fast and repeatable, but limited when quality needs interpretation.
LLM-as-a-judge
Best for rubric-based judgments such as relevance, grounding or instruction adherence. Score dimensions separately and calibrate against expert-reviewed examples. A fluent explanation is not proof of correctness.
Agent-as-a-judge
Best when assessment needs investigation across steps or tools. A bounded evaluator inspects the action history and verifies outcome state. Restrict its access and budget, preserve evidence and check its conclusions independently.
Human-as-a-judge
Best for ambiguous requirements, domain nuance, rubric creation and disputed results. Train reviewers with examples, measure agreement and adjudicate differences rather than silently averaging them.
When to use each
Is it objectively checkable?
Use code. If a rule, value or state change can be verified exactly, an AI judge only adds cost and noise.
Does it need interpretation at scale?
Use a calibrated LLM judge, with a written rubric, reference examples and measured agreement with experts.
Is the stakes or ambiguity high?
Use human review, and use those decisions to improve the rubric and the automated judges over time.
Beyond the judge
Repeated-run testing
Run selected scenarios multiple times with recorded configuration. Report how often the agent succeeds on every attempt, not just whether it succeeded once.
Simulation and fault testing
Simulated users for multi-turn journeys, plus delayed or malformed tool responses and unavailable dependencies, to test recovery in a controlled environment.
Metamorphic testing
Rephrase, reorder or vary inputs in ways that should not change the right outcome, and check that the agent's behavior stays correct.
Pairwise comparison
Compare candidate and baseline outputs on the same tasks with randomized order and a calibrated rubric, to support change decisions.
Outcome verification
Inspect the resulting task state directly. Confirm that the promised action occurred and that no required step was skipped.
Trajectory review
Examine the sequence of reasoning steps and tool calls, not only the endpoint, to find wasted steps, unsafe shortcuts and lucky successes.
Metrics that mean something
Success with denominators
Every rate reports what was attempted, what passed and the explicit success rule. A percentage without a denominator is not evidence.
Consistency, not just capability
"Succeeds at least once in k tries" and "succeeds on all k tries" answer different questions. Customer-facing agents usually need the second.
Severity over averages
One critical failure can outweigh many small wins. We report severe failures separately so an aggregate score never hides them.
Measuring hallucinations
Claim-level verification
Split each answer into individual claims and check each against approved sources, tool results or the conversation. One invented detail fails the answer.
Answerable and unanswerable cases
Include questions your sources can't answer. Score honest abstention as correct, so the evaluation doesn't reward confident guessing.
Action claims checked against state
Every "done" statement is matched to real tool calls and resulting records. This is how action hallucinations are caught.
Our evaluation workflow
Plan and curate
Define scope and success criteria. Review test cases, expected results and coverage with your business expert.
Calibrate and execute
Validate evaluators against reviewed examples, then run the agreed suite with reproducible version and environment details.
Analyze and iterate
Investigate disagreements, prioritize failures, retest fixes and update the dataset without contaminating the held-out benchmark.
Honest limits
Coverage is never complete
Tests sample possible behavior. We state what was covered and what wasn't, so conclusions aren't stretched beyond the evidence.
Judges make mistakes
Every evaluator, including humans, has error rates. We measure them and report them alongside results.
Standards are still forming
Stanford's AI Index has highlighted the lack of standardized evaluations across the industry. We align with established guidance such as the NIST AI Risk Management Framework without claiming certification.
Make your next decision with evidence.
Start with one workflow, clear success criteria, and an evaluation plan your team can use.