Why Do AI Agents Hallucinate? Causes, Risks, and What to Do

AI agents invent facts, policies, and even actions, and say it all with confidence. Here's why it happens and how to catch it before your customers do.

A customer asks your support agent about a refund for a cancelled trip. The agent answers quickly, politely, and with total confidence, describing a refund policy your company doesn't have.

It is the failure business leaders worry about most, and with good reason. In agents that can take actions, invented information doesn't just mislead people.

This post explains why AI agents hallucinate, why agents raise the stakes compared with chatbots, and what you can realistically do about it.

A confident answer that wasn't true

Two real cases show what's at stake.

In Moffatt v. Air Canada (2024), a Canadian tribunal considered a chatbot that told a grieving customer he could apply for a bereavement fare discount after travelling. The airline's actual policy said otherwise. The tribunal held Air Canada responsible for what its chatbot said.

In Mata v. Avianca (2023), lawyers in New York filed a brief citing court decisions that a chatbot had invented. The cases didn't exist. The court sanctioned the lawyers.

Neither failure was dramatic in the moment. Both answers looked right. That's what makes hallucinations dangerous: they arrive in the same fluent, confident tone as correct answers.

Why AI models hallucinate

Language models don't look facts up the way a database does. They generate the most plausible next words based on patterns learned from huge amounts of text, plus whatever information you give them at the time. Usually that produces useful, accurate output. Sometimes the most plausible-sounding answer isn't the true one.

Researchers at OpenAI argued in 2025 that part of the problem lies in how models are trained and evaluated: many benchmarks reward a confident guess more than an honest "I don't know". A model that guesses scores better on those tests than one that admits uncertainty, so models learn to guess.

The US National Institute of Standards and Technology lists this behavior, which it calls confabulation, as one of the core risks of generative AI in its Generative AI Profile of the AI Risk Management Framework.

What triggers hallucinations in practice

In real agents, hallucinations cluster around a handful of triggers. Knowing them tells you where to test.

- Missing knowledge. The answer isn't in your documents or systems, so the agent fills the gap with something plausible.
- Outdated or conflicting sources. Two policy versions are both searchable, or an old price list is still indexed. The agent picks the wrong one or blends them.
- False-premise questions. "Since you offer free returns after 90 days…" Agents often go along with a user's incorrect assumption instead of correcting it.
- Failed or empty tool results. A lookup times out or returns nothing. Rather than say so, the agent invents a result to keep the conversation moving.
- Long conversations. Details given early get distorted or contradicted many turns later.
- Guessing rewarded over "I don't know". If prompts, training or your own tests discourage the agent from admitting uncertainty, it will guess.

Why agents raise the stakes

A chatbot hallucination is a wrong answer. An agent hallucination can become a wrong action. Agents add failure types chatbots don't have:

- Action hallucinations. The agent tells the customer the booking was changed, but the tool call never happened or failed.
- Tool and argument hallucinations. The agent invents an order number, an email address or an amount and passes it to a real system.
- Invented commitments. Discounts, exceptions or deadlines your business never offered, which customers may then hold you to.

Specialist products aren't immune either. A Stanford study of AI legal research tools, including products marketed as reducing hallucinations, still found hallucinated or incorrect answers on a meaningful share of test queries. Domain focus and retrieval reduce the risk; they don't remove it. That's why every agent needs testing on its own work.

What evaluation can and can't do

No honest provider can promise zero hallucinations with today's models. What evaluation can do is substantial:

- Measure how often your agent hallucinates, by workflow and type.
- Locate the causes: retrieval gaps, outdated content, prompt instructions, model behavior or tool handling.
- Test fixes fairly, one change at a time on the same test cases.
- Set acceptable thresholds for each workflow and route the riskiest cases to people.

How to test for hallucinations

Good hallucination testing uses several techniques together:

1. Source-anchored test cases. Pair each question with the approved source that should answer it and an expert-reviewed reference answer.
2. Questions with no answer. Deliberately include requests your sources can't answer. Saying "I don't know", asking a question or handing off is a *pass*. Inventing an answer is a fail.
3. Claim-level checking. Break answers into individual claims and check each one. A good answer with one invented detail should still fail.
4. Action verification. For every "done", check the tool calls and the resulting record.
5. Repeated runs. Hallucinations come and go, so run key cases several times and measure a rate.

Choosing the right judge

Not every check needs an expensive judge. Climb the ladder only as far as you need to.

- Code checks for exact facts: IDs, prices, dates and amounts against your systems.
- A calibrated LLM judge for paraphrased claims: is this sentence supported by the retrieved policy?
- An agent judge for claimed actions: did the system state actually change?
- Human review for ambiguous, high-stakes or new cases, and to calibrate the automated judges.

What this means for your role

VPs and executives. Don't ask "does it hallucinate?" (it can). Ask "how often, in which workflows, and what happens when it does?" Set an acceptable rate for each workflow based on the cost of an error.

Directors. Make hallucination rate a release metric that is tracked across versions, alongside task success and cost.

Managers. Build the unanswerable test set first. It's the fastest way to see whether your agent guesses. Make sure your knowledge sources have clear owners and that outdated versions are removed.

HR and talent leaders. Domain experts who review agent answers are your best defense. Define the reviewer role, give them time and training, and recognize the work.

Checklist

- List the facts your agent must never get wrong (prices, policies, eligibility, deadlines).
- Add 10–20 questions your sources can't answer, and score "I don't know" as correct.
- Check every claimed action against the real system state.
- Remove or clearly retire outdated documents from retrieval.
- Run key scenarios at least three times and report a rate, not an anecdote.
- Have a domain expert review a sample of passes as well as failures.

Key takeaways

- Hallucinations are plausible guesses presented with confidence; incentives that reward guessing make them worse.
- Agents raise the stakes: invented information can drive actions, commitments and records.
- Common triggers are missing or outdated knowledge, false premises, failed tools and long conversations.
- You can't reach zero, but you can measure, locate causes, test fixes and set thresholds.
- Combine judges: code for exact facts, calibrated LLM judges for claims, agent checks for actions, humans for judgment.

How Test2 helps

Our AI Agent Hallucination Testing & Evaluation work measures how often your agent hallucinates, where and why, and tests which fixes actually reduce it. It runs within our AI Agent Evaluation Assessment, implementation and managed evaluation engagements.

Sources

- Moffatt v. Air Canada, 2024 BCCRT 149.
- Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y. 2023).
- OpenAI, [Why language models hallucinate](https://openai.com/index/why-language-models-hallucinate/) (2025).
- NIST, [Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1)](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf) (2024).
- Magesh et al., *Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools*, Stanford RegLab and HAI (2024).