An agent can quote the right refund policy and still invent an exception. The business problem is the unsupported commitment a customer may act on. Before changing the model, create an evaluation that can distinguish a supported answer from a plausible invention.
Promptfoo can help run that evaluation. Its llm-rubric assertion lets you define a judgment in plain language. The difficult work is supplying the right evidence and checking whether the judge applies your rule consistently.
Why can a confident answer be wrong?
OpenAI's research on hallucinations describes how common evaluation incentives can reward guessing instead of acknowledging uncertainty. That helps explain one cause of confident errors; it is not a diagnosis of your entire agent.
In a deployed workflow, missing documents, obsolete policies, incorrect retrieval and misread tool results can also produce unsupported answers. You need cases that separate these conditions. A single score called “accuracy” will not explain which fix to try.
Consider a fictional support policy: unused products can be returned within 30 days. A customer asks to return an unused product after 60 days. The agent says a manager can always approve it. Nothing in the approved policy grants that exception. A helpful tone should not earn a pass.
Define one judgment before collecting scores
For this article, the feature is a source-grounded llm-rubric evaluation. It judges whether material claims are supported by the supplied policy. It does not establish whether a refund actually happened; that requires tool and system-state checks.
Use three reviewed case types:
| Case | Evidence | Expected behavior |
|---|---|---|
| Answerable | Policy clearly permits a return | Explain the permitted option accurately |
| Unanswerable | Policy does not address the request | Acknowledge the gap and ask or escalate |
| Conflicting | Two policy versions disagree | Follow a defined precedence rule or flag uncertainty |
This design prevents a lazy evaluator from rewarding refusal on every question. An agent must answer when evidence supports an answer and acknowledge uncertainty when it does not.
Build the first evaluation in six steps
- Choose one workflow and decision, such as whether a revised support prompt is ready for review.
- Create a small, expert-reviewed set. Twenty cases is a practical workshop starting point, not proof of production coverage.
- Store the approved source text and expected behavior with each case. Record policy dates and versions.
- Capture the actual agent answer through your existing integration or a reviewed provider adapter. Merely running a generic chat model does not test your deployed agent.
- Write a rubric that fails unsupported material claims, accepts supported paraphrases and distinguishes justified uncertainty from evasion.
- Compare judge decisions with independent expert labels before using the score for a release decision.
The Promptfoo rubric documentation describes the assertion and grader configuration. Explicitly select the judge and passing rule instead of relying on an environment-dependent default. A threshold is a scoring policy, not a probability that the agent is correct.
Report a rate that means something
Choose a unit and keep it consistent. An answer-level rate counts answers with at least one unsupported material claim. A claim-level rate counts individual unsupported claims. Those denominators answer different questions.
Illustrative arithmetic only: if experts identify unsupported material claims in 8 of 40 reviewed answers, the observed answer-level rate is 20%. It is not a forecast of all production traffic. Report the selected cases, repeated runs, missing evidence and judge disagreements alongside it. A rare but severe invented commitment deserves separate attention even when the average improves.
Plan time, talent and the next decision
A manager needs a policy owner and an engineer who can capture the actual answer and its evidence. HR or L&D can help reserve reviewer time and teach consistent labeling. For planning only, 20 cases at three minutes per review require 60 reviewer-minutes per reviewer, before discussion and corrections. Setup time depends on access and source quality; it is not included in that arithmetic.
The VP's useful output is a bounded finding: where the agent makes unsupported claims, what those failures mean and which changes should be tested next. Source cleanup, retrieval changes, instructions and model selection are hypotheses to compare against the same reviewed cases. Evaluation measures whether a fix helped; it does not automatically make the fix.
