Verify the Artifact, Not the Claim.
An agent says, “The customer report is ready.” The file exists, but it contains the wrong reporting period. Reading the final chat response cannot establish whether the job was completed correctly.
A tool-capable evaluator can inspect the artifact behind the claim. Promptfoo's agent-rubric provides an agentic grading option for that kind of evidence-seeking evaluation. It should be a bounded investigation with explicit evidence, not an unrestricted second agent whose confident opinion becomes the verdict.
The grader needs the relevant artifact and a clear acceptance rule.
Use a code check when it can answer the question
If the only question is whether a file exists at a known path, use code. If the question is whether a field equals a known value, use code again. Those tests are usually easier to reproduce and inspect.
An agent judge becomes useful when evaluation requires investigation across several observable items: finding the requested artifact, reading its relevant sections and comparing them with an agreed task requirement. Even then, combine it with exact checks where possible. The evaluator's extra flexibility adds runtime, model usage and another source of error.
Make an evidence contract before configuring the grader
For a fictional monthly report, the contract might contain:
| Requirement | Evidence | Verdict if unavailable |
|---|---|---|
| Correct customer | Artifact heading and reviewed request | Unable to verify |
| Correct period | Report dates and requested period | Fail if inconsistent |
| Requested sections | Actual report contents | Fail if missing |
| File delivered | Approved delivery receipt | Unable to verify without receipt |
A file in a workspace does not prove delivery. A paragraph saying “all figures were checked” does not prove calculation correctness. Write these distinctions down before asking the judge to investigate.
Configure access deliberately
The Promptfoo Agent Rubric documentation describes a coding-agent grader and explains that its default isolated workspace does not expose your project files. To inspect a supplied artifact, the evaluator must be explicitly given the appropriate workspace or evidence.
That matters operationally: adding agent-rubric to a test is not automatic access to a CRM, transaction system or the agent's environment. A customer-approved adapter or fixture process must supply the evidence. Check the installed framework and grader-runtime versions before implementing it.
For a first pilot, use reviewed copies of generated artifacts in a dedicated fixture workspace. Give the grader only what the evaluation needs. Preserve the artifact version and a mapping from each case to its evidence. The purpose is functional outcome evaluation, not code scanning or a security audit.
Run a bounded investigation
- State the user's intended outcome and acceptance requirements.
- Identify which artifact and evidence belong to the case.
- Ask the grader to report each requirement as met, not met or unable to verify, with an observable evidence reference.
- Bound the investigation's time, tool access and usage budget.
- Treat missing evidence as a separate result, not success.
- Independently inspect a sample of passes and every material failure before making a release recommendation.
Do not ask for hidden reasoning as proof. A file reference, relevant excerpt or verified value is something another reviewer can check; an eloquent explanation alone is not.
Evaluate the evaluator before expanding the pilot
Create fixtures with known defects: a missing section, a wrong date and an artifact that exists under the wrong case. Include a correct artifact and an evidence-missing case. Observe whether the evaluator distinguishes them. Check exact facts with code when available.
Report false passes, false failures and unable-to-verify cases separately. Compare its incremental useful findings with a simpler code check and a human review. If it adds cost without finding meaningful failures, keep the simpler approach.
A director owns the evidence boundary and required decision. The engineer connects fixtures and grader runtime. The domain reviewer checks semantic conclusions. HR or L&D can plan training in evidence review rather than assuming any fluent reader can adjudicate the output.
For planning only, a ten-artifact pilot reviewed for six minutes per artifact requires one reviewer-hour, excluding setup and adjudication. Measure actual grader runtime and usage; do not assume a fixed cost per investigation.
