| NIST officially calls AI hallucination confabulation in its July 2024 Generative AI Profile. OpenAI’s own research ties the cause to training incentives that reward guessing over admitting uncertainty. Hallucination rates range from roughly 1 percent on grounded summarization to over 60 percent on ungrounded legal and medical tasks. Retrieval-augmented grounding is the most consistent detection and mitigation method. Courts have documented over 120 real cases of AI-hallucinated citations, including one involving Anthropic’s own Claude. |
LLM hallucination is the confident generation of false information that sounds plausible but is not grounded in fact, a problem NIST’s official AI risk framework calls confabulation. OpenAI’s own research traces the cause to training and evaluation procedures that reward guessing over acknowledging uncertainty, not a mysterious flaw unique to any one model. Hallucination rates vary enormously depending on the task, from around 1 percent on grounded summarization to over 60 percent on ungrounded legal or medical work. This article covers why it happens, how to detect it, and which models hallucinate most across the benchmarks that actually measure it.
LLM Hallucination Defined: NIST Calls It “Confabulation” in Its July 2024 AI Risk Framework
NIST’s AI 600-1 Generative AI Profile, published July 26, 2024, gives large language model hallucination its official United States government name: confabulation. NIST defines it as the production of confidently stated but erroneous or false content by which users may be misled or deceived.
Confabulation sits alongside eleven other named risks inside the broader NIST AI Risk Management Framework, a framework organizations increasingly use to structure how they govern generative AI deployments. Listing hallucination as a defined, government-named risk category matters because it moves the conversation away from an informal complaint about AI being unreliable and toward a measurable, governable problem with its own classification.
That distinction matters for how this article treats the term. A hallucination is not simply a wrong answer. It is specifically the confident, plausible-sounding presentation of false content, delivered with the same tone and structure a model uses for a correct answer, with no built-in signal telling the reader anything is wrong. NIST’s AI 600-1 Generative AI Profile documents the full definition and its place within the wider risk framework.
Why AI Hallucinates: OpenAI’s Own Research Ties It to Training Incentives That Reward Guessing
OpenAI’s own research paper on the subject argues that language models hallucinate because standard training and evaluation procedures reward confident guessing over honestly acknowledging uncertainty. The paper backs that claim with a formal mathematical link between a model’s hallucination rate and its underlying misclassification rate, the same kind of error rate studied in ordinary supervised learning.
The mechanism works like this. During pretraining, a model learns to predict the next word in a sequence. Common patterns like grammar and spelling improve reliably with scale, since they repeat constantly across training data. Arbitrary, low-frequency facts, the kind that only appear once or twice in the entire training set, cannot be predicted from patterns the same way, and that gap is where hallucination originates. Post-training evaluation is meant to correct this, but the paper argues it often makes the problem worse instead, because most benchmarks score an honest “I don’t know” the same as a confidently wrong answer. That scoring choice teaches a model that guessing is always at least as good as abstaining, and often better.
OpenAI’s paper includes a direct demonstration of this. Researchers asked a widely used chatbot for the title of a real person’s PhD dissertation and received three different, entirely incorrect answers. Asked for that same person’s birthday, the model produced three different wrong dates.
Reasoning ability does not automatically fix this. OpenAI has disclosed that its own o3 and o4-mini models hallucinated at 33 percent and 48 percent, respectively, on a public-facts benchmark, notably higher than an earlier system, showing that deeper reasoning and lower hallucination do not move together in a straight line. The paper’s own proposed fix is to change evaluation culture itself, rewarding appropriate uncertainty in benchmarks and leaderboards instead of penalizing it. OpenAI’s own research on why language models hallucinate lays out the full argument and its supporting proof.
Which Models Hallucinate Most: Benchmark Scores From 1% to 48% Depending on the Task
Hallucination rate is not one number. Grounded summarization benchmarks put frontier 2026 models near 1 to 2.5 percent, retrieval-augmented generation faithfulness benchmarks put the same class of models at 4 to 9 percent, and ungrounded legal or medical tasks push some tools past 60 percent. Reading any single hallucination statistic without knowing which of these tasks it measures gives a misleading picture.
Hallucination Rate by Benchmark Type
| Benchmark | What It Measures | Typical 2026 Frontier Range |
| Vectara HHEM (summarization) | Factual consistency when summarizing short documents | Roughly 1.0% to 2.5% |
| RAGTruth (RAG faithfulness) | Faithfulness to retrieved source material | Roughly 4% to 9% |
| Independent 2026 frontier cohort study (package hallucination) | Fabricated references across a large prompt set | 4.62% to 6.10% across tested models |
| Stanford HAI (legal-domain queries) | Accuracy on real legal research questions | 1 in 6 or more queries affected |
| Ungrounded medical summarization (academic study context) | Fabricated clinical content without retrieval grounding | Up to 60% |
These five rows should never be read as a single ranking. Each benchmark tests a different task, a different domain, and a different definition of what counts as a hallucination, and no model in this comparison scores zero on any of them. Reasoning depth and hallucination rate also do not move in one consistent direction. OpenAI’s own disclosure that o3 and o4-mini hallucinated at 33 percent and 48 percent, covered in the previous section, shows that a more capable reasoning model is not automatically a more truthful one on every task.
How to Detect AI Hallucination: Retrieval-Augmented Grounding and Cross-Model Verification
Retrieval-augmented grounding is the most consistent detection and mitigation method available, reducing hallucination rate by anchoring a model’s answer to retrieved source text instead of relying on unaided next-word prediction. Grounded generation gives a model a specific passage to work from and evaluates its answer against that passage directly, rather than testing open-ended recall from memory.
Cross-model verification is a practical technique worth pairing with grounding rather than replacing it. Asking a second, different model to verify a citation, date, or fact catches a meaningful share of errors, since two separate models built on different training data rarely hallucinate the same wrong answer in the same way. When two models disagree on a fact, that disagreement itself is a useful signal to investigate further before publishing or acting on the answer.
Explicit instruction framing also measurably reduces hallucination in practice. Telling a model directly not to invent facts, and to state plainly when an answer is not present in the given source material, shifts the model away from the guessing behavior OpenAI’s research identifies as the root cause.
None of this eliminates hallucination. RAGTruth’s own reported range of 4 to 9 percent shows that even grounded generation still produces a meaningful failure rate. Grounding and verification reduce the problem substantially. They do not solve it.
AI Hallucination in Court: 120+ Documented Cases of Fake Legal Citations, Including Anthropic’s Own Claude
A public database tracking real court cases has documented more than 120 instances of AI-hallucinated legal citations reaching actual court filings, a count the researcher who compiled it describes as a floor rather than a ceiling, since cases that never draw a judge’s attention are not counted at all. Stanford HAI’s own research found that legal AI tools hallucinate on 1 in 6 or more benchmarking queries, a rate high enough to make unverified use in legal work a genuine professional risk rather than a rare edge case.
One documented case stands out. In May 2025, a lawyer representing Anthropic used an incorrect legal citation, with an inaccurate title and inaccurate authors, that Claude had generated during Anthropic’s own ongoing litigation. Anthropic’s team acknowledged the error directly, calling it an honest citation mistake rather than a fabrication of authority, and apologized to the court. A leading AI lab’s own model hallucinating inside that same lab’s own case is one of the clearest illustrations available that no current model, regardless of who builds it, is immune to this problem.
The pattern documented across the tracked cases has also shifted over time. Earlier cases involved mostly self-represented litigants without legal training. More recent cases increasingly involve practicing lawyers and their firms, several of whom have faced monetary court sanctions once the fabricated citations were discovered. Medical hallucination carries a different but comparably serious stake. Academic research on ungrounded medical summarization found rates as high as 60 percent, a domain where a single fabricated detail carries consequences well beyond a rejected legal filing.
NIST’s Generative AI Risk Framework: How Confabulation Fits Into Govern, Map, Measure, Manage
NIST’s AI Risk Management Framework organizes confabulation and eleven other generative AI risks under four functions: Govern, Map, Measure, and Manage. That structure gives organizations a way to treat hallucination as a governable, measurable category instead of an unpredictable side effect to simply tolerate.
Govern covers the policies and acceptable-use standards an organization sets before deployment. Map covers identifying where hallucination risk actually applies within a specific use case, since a customer support chatbot and a legal research tool carry very different exposure. Measure covers direct monitoring of confabulation rate itself, alongside the other trustworthy AI characteristics NIST names, including validity, reliability, and explainability. Manage covers response prioritization, ongoing testing, and disclosure protocols for material confabulation incidents once they are identified.
Framing hallucination this way changes the practical question from whether an AI system is reliable to how a specific deployment is measured, monitored, and managed for confabulation risk, a meaningfully more useful question for anyone actually responsible for how an AI system gets used.
LLM Hallucinations’ Place in the Wider Large Language Models Subcategory
Hallucination risk applies to every model discussed elsewhere in this subcategory, from DeepSeek V4 Pro and GPT-5.6 Sol to Claude Opus 5 and Gemini 3.1 Pro. No benchmark comparison of raw capability is complete without also asking how often a model states something false with full confidence, since coding speed, reasoning depth, and writing quality all matter less if the underlying output cannot be trusted without verification.
This risk also extends past large language models on their own. LLM-generated content that reads as confident and well-formed is exactly the kind of output that can evade automated filters when it shows up somewhere unexpected, such as phishing content built to slip past spam detection. That connection sits in a different part of AI News and Tech coverage and will be linked directly once that coverage is published.
Universalnest’s LLM Coverage: the Definition Guide and the Model Comparisons
This explanation of hallucination builds on the same token, context, and benchmark fundamentals covered across Universalnest’s introduction to large language models, which explains how context windows and reasoning benchmarks work before naming specific models.
For the same benchmark discipline applied to DeepSeek V4 Pro and GPT-5.6 Sol’s pricing and coding comparison, see Universalnest’s most recent model comparison. For how GPT-5.6 Sol, Claude Opus 5, and Gemini 3.1 Pro compare on reasoning, coding, and writing benchmarks, see the article before it. Both comparisons carry the same caution this article makes explicit: a benchmark score is never the complete picture, and hallucination rate is one of the reasons why.
Frequently Asked Questions
What is the difference between an AI hallucination and a normal factual error?
A normal factual error can include hedging or acknowledged uncertainty. A hallucination is specifically confident, plausible-sounding false content presented with no signal that anything is wrong, which is why NIST classifies it as its own named risk category rather than general inaccuracy.
Does retrieval-augmented generation eliminate hallucination?
No. Retrieval-augmented generation reduces hallucination substantially by grounding answers in retrieved source text, but RAGTruth’s own benchmark still shows a 4 to 9 percent failure rate on faithfulness tasks. Grounding lowers the risk considerably. It does not remove it entirely.
Which hallucination benchmark should actually be trusted?
No single benchmark should be trusted alone. Vectara HHEM measures summarization groundedness, RAGTruth measures RAG faithfulness, and legal or medical studies measure domain-specific accuracy. Match the benchmark to your actual task rather than relying on one general hallucination percentage.
Can reasoning models still hallucinate even with extended thinking?
Yes. OpenAI has disclosed that its own o3 and o4-mini reasoning models hallucinated at 33 percent and 48 percent, respectively, on a public-facts benchmark, notably higher than an earlier, less reasoning-focused system, showing extended thinking does not automatically lower hallucination rate.
What is NIST’s official term for AI hallucination?
NIST’s AI 600-1 Generative AI Profile, published July 26, 2024, officially names the phenomenon confabulation, defined as the production of confidently stated but erroneous or false content that may mislead or deceive users relying on it.
Has an AI company ever been directly involved in a court hallucination incident?
Yes. In May 2025, a lawyer representing Anthropic used an incorrect citation that Claude had generated during the company’s own litigation. Anthropic’s team acknowledged the error and apologized to the court, calling it an honest mistake rather than a fabrication.
Do larger models hallucinate less than smaller ones?
Not consistently. Benchmark data shows hallucination rate depends more on task type and grounding than raw model size, and OpenAI’s own o3 and o4-mini disclosures show newer, more capable reasoning models can hallucinate more often than earlier, smaller systems on certain tasks.
