Invented facts are only the start. Nine ways AI agents hallucinate, from fake policies to actions that never happened, and how to test for each one.
"Our agent hallucinates sometimes" is too vague to act on. Which hallucinations? In which workflows? With what consequences?
You can't measure, prioritize or fix what you can't name. This post sets out **nine types of AI agent hallucination**, each with an illustrative example, why it matters and how to test for it. Several of them barely exist in simple chatbots; they appear when AI starts taking actions.
If you're new to the topic, start with Why Do AI Agents Hallucinate?
1. Fabricated facts
What it is: a statement with no basis in your data or the real world, such as a product feature, a price, a date or a figure the agent made up.
Example: asked whether a plan includes phone support, the agent says "Yes, 24/7 phone support is included." It isn't.
Why it matters: customers act on what they're told, then contact support or demand what was promised.
How to test: pair questions with the source that answers them and check the facts in the answer. Use code checks for exact values (prices, plan names) against your product data.
2. Unsupported or contradicted answers
What it is: the agent retrieved the right document, but its answer adds details, omits conditions or contradicts what the source says.
Example: the policy says refunds are available "within 30 days *with receipt*." The agent says "within 30 days."
Why it matters: this is the most common hallucination in knowledge assistants, and the hardest to spot because the answer is *mostly* right.
How to test: break answers into individual claims and check each against the retrieved source with a **calibrated LLM judge**. Look specifically for dropped conditions and exceptions.
3. Invented policies and commitments
What it is: discounts, exceptions, deadlines or rules your business never offered.
Example: "As a loyal customer, I can waive the change fee for you." No such policy exists.
Why it matters: customers may hold you to it. In *Moffatt v. Air Canada* (2024), a tribunal held the airline responsible for a fare policy its chatbot described incorrectly.
How to test: write test cases that invite exceptions ("I've been a customer for ten years…", "My flight was cancelled because of weather…") and check every offer against your actual policy rules. Have a policy owner review a sample.
4. Action hallucinations
What it is: the agent says it did something it didn't do, or that failed.
Example: "Your booking is now changed to Friday." The change tool returned an error, so nothing changed.
Why it matters: this is arguably the most damaging agent-specific type. The customer leaves believing a task is complete and finds out later, when it's harder and costlier to fix.
How to test: for every claimed action, **check the tool calls and the resulting record** in a test system. A reply-only judge will never catch this. Use code checks on state, or an **agent-as-a-judge** for complex multi-step tasks.
5. Tool and argument hallucinations
What it is: the agent calls a tool that doesn't exist, or fills real tool calls with invented values: an order number, an email address, an amount or a date the user never provided.
Example: the user says "cancel my last order"; the agent passes an order ID it guessed rather than looking one up.
Why it matters: invented arguments turn a wording problem into a **data** problem: wrong records changed, wrong customers contacted.
How to test: validate tool names and arguments with code. Check that every argument value came from the conversation, a lookup or a tool result, not thin air. Include cases where required information is missing; the right behavior is to ask.
6. Fabricated citations and references
What it is: links, document titles, case names or quotations that look real but don't exist, or don't say what the agent claims.
Example: "According to section 4.2 of our Terms of Service…", where section 4.2 says something else entirely.
Why it matters: citations signal authority, so fake ones are especially persuasive. In *Mata v. Avianca* (2023), lawyers were sanctioned after filing a brief with court cases invented by a chatbot.
How to test: check that every cited source exists (code) and that it supports the claim attached to it (LLM judge plus spot-checks by an expert).
7. Context and memory errors
What it is: misremembering what the user said earlier, mixing up details between customers or sessions, or "recalling" a conversation that never happened.
Example: early in a chat the customer says the delivery address is their office; ten turns later the agent confirms delivery to their home.
Why it matters: long conversations are where real work happens, and errors there are easy to miss in short tests.
How to test: write multi-turn test conversations where key details appear early and must be used correctly later. Include changed requests ("Actually, make that Thursday").
8. Capability overclaims
What it is: the agent promises to do something it can't do.
Example: "I've emailed you a copy of your invoice," from an agent with no email capability. Or "I'll call you back tomorrow."
Why it matters: customers wait for something that never comes, then escalate frustrated.
How to test: list what the agent can and can't do, then write requests that tempt it to overreach. Check promises against its real tools.
9. Numeric and calculation errors
What it is: totals, balances, prorated amounts, conversions and dates that look precise but are wrong.
Example: the agent calculates a prorated refund of $47.83 when the correct amount is $38.47.
Why it matters: precise-looking numbers are trusted, and errors in quotes, invoices or refunds have direct financial and compliance consequences.
How to test: compute expected values in code and compare exactly. Where possible, have the agent use a calculation tool rather than doing the arithmetic itself, then test that it does.
Which types matter most for your agent?
Not every type carries the same risk for every agent. A research assistant worries most about citations; a booking agent about actions and policies; a billing agent about numbers. Place each type by business impact and how often it could occur in your workflows.
The matrix above is illustrative. For most customer-facing agents that take actions, action hallucinations, invented policies and tool argument errors belong in the top-right corner: test them thoroughly before launch and keep them in every regression run.
What this means for your role
VPs and executives. Ask for hallucination results broken down by type and workflow. A single overall "hallucination rate" can hide the one type that creates legal or financial exposure.
Directors. Adopt a shared taxonomy across teams so results are comparable and fixes can be reused.
Managers. Build a few targeted test cases for each high-priority type; 5–10 per type is a useful start.
HR and talent leaders. Different types need different reviewers: policy owners for type 3, finance for type 9, support leads for type 7. Involve the right experts and budget their time.
Checklist
- Identify the three types that matter most for your agent.
- Write 5–10 targeted test cases for each.
- Verify every claimed action against system state.
- Compute expected numbers in code.
- Tag every confirmed failure with its type.
- Report hallucination rates by type and workflow.
Key takeaways
- Nine types: fabricated facts, unsupported answers, invented policies, action hallucinations, tool argument errors, fake citations, context errors, capability overclaims and numeric errors.
- Agents add new types that turn wrong words into wrong actions and wrong data.
- Each type needs its own test, and often its own judge.
- Prioritize by business impact for your specific agent.
- Name it to measure it: tag failures and report rates by type.
Sources
- *Moffatt v. Air Canada*, 2024 BCCRT 149.
- *Mata v. Avianca, Inc.*, No. 22-cv-1461 (S.D.N.Y. 2023).
