Why generative AI fails exactly where you don't notice
A confident confabulator
AI hallucinations aren't random — they cluster, systematically and predictably, in the topics you cannot independently verify.
Ask an Artificial Intelligence Model about the French Revolution, and you get a competent summary. Ask about a minor 18th-century poet, and you'll still get a competent-sounding summary — except this one might be partly invented, with fabricated quotes and a biography stitched together from similar figures. Both answers are provided with the same polished authority of an expert in the field. One is real, one is mostly fiction.
This isn't a random failure; AI hallucinations are not evenly distributed across topics. They cluster, systematically and predictably, in exactly those areas where users know less and are thus least able to detect them. This makes them one of the biggest problems of AI.
So in short: the more thoroughly a subject has been written about and made available to the AI, the more reliably a model can talk about it. The further a query drifts into specialist territory, the more the output shifts from recall to invention. This is not a calibration problem that better training will solve — it's intrinsic to how these systems work. The pattern shows up consistently wherever it's been measured and is confirmed by many studies.
About this post
We examine why AI hallucinations concentrate in topics where users are least equipped to notice them — and why this structural pattern makes them a usability problem, not just a technical one.
Key takeaways
Hallucination rates track inversely with training data density. Topics with little data produce both higher error rates and lower user detection. Skepticism needs to scale inversely with your own expertise on a subject.
The original ideas, structure, and much of the language are human-created, but AI was used to develop, enrich, or rework portions of the content — for example, researching sources, rewriting sections for clarity, or expanding on arguments - please click for more information
"The failures will be in cases where it's harder for a reader to notice — because they are more obscure."
Emily M. Bender, University of Washington
The table below shows some information on hallucination levels across multiple topics. The results show that hallucinations are negatively correlated with domain knowledge - the more information is available, the less AI models confabulate.
| Domain / Task | Hallucination Rate | Source |
|---|---|---|
| Short-document summarization (top models) | 0.7–1.5% | Vectara HHEM Leaderboard |
| Citations on well-studied medical topic (major depression) | ~6% | Deakin University, 2025 |
| Citations on less-studied medical topic (body dysmorphic disorder) | ~29% | Deakin University, 2025 |
| Real-world conversational benchmark | 31.4% | AuthenHallu, arXiv:2510.10539, 2025 |
| Purpose-built legal AI tools (Lexis+, Westlaw) | 17–34% | Stanford RegLab / HAI, 2024 |
| General LLMs on specific legal queries | 58–88% | Stanford "Hallucinating Law", 2024 |
| Code generation referencing non-existent libraries | up to 99% | LLM Hallucination Statistics, 2026 |
Sources: Vectara Hallucination Leaderboard (github.com/vectara/hallucination-leaderboard); Deakin University, ChatGPT citation accuracy in mental health literature reviews, 2025; Stanford HAI, "Hallucinating Law" and "Hallucination-Free?", 2024; AuthenHallu benchmark, arXiv:2510.10539, October 2025.
The core problem: How AI stores and retrieves
The mechanism behind this relationship is not mysterious — it's directly baked into how language models process language in the first place. There is no actual recognition of content, just a very refined vector comparison that finds the nearest possible match. If there are many finds on a subject, "near" means that it is almost always correct. If there is limited source material, "near" can be too far away. Once we understand this fundamental concept of how a query actually moves through the system, the cumulation of hallucinations in less-covered subjects is a logical consequence.
This all applies to us humans too. Each of us has areas of expertise where we can confidently state facts and rarely make mistakes and others where our knowledge is limited.
How Language Models see text
Text never enters or leaves a language model as text. It gets broken into tokens — subword fragments, each mapped to an integer ID — and each token is then converted into a high-dimensional vector, typically 1,000 to 10,000 numbers long. That vector is everything the model "knows". Training adjusts billions of parameters so that tokens appearing in similar contexts end up with mathematically similar vectors, and so the model can predict which vector is most likely to come next.
At no point does the system check anything against truth. It checks proximity. The sentences "penicillin was discovered by Alexander Fleming" and "penicillin was discovered by Alexander Flemming" are nearly identical as vectors — the model outputs whichever pattern the training data made statistically more likely., not whichever is factually correct. Where training data is thin, the nearest statistical neighbor is often too far away to represent a fact, and the model presents the wrong answer dressed in the right shape - with confidence.
The key difference is the confidence with which uncertainty is presented. For AI systems, confidence is baked into their generation logic, at it is not connected to the validity of the content they are presenting. They sound authoritative and certain when making things up.
This confidence mismatch compounds the problem. Humans use linguistic confidence as a heuristic for reliability — nervous, hedging speech signals uncertainty; fluent assertion signals knowledge. Language models break this cue. They speak with uniform fluency whether they're on solid ground or free-associating from fragments, and the signal humans rely on to gauge speaker reliability is, for practical purposes, absent from their output. And most LLM companies decided that users don't like a "sorry, I can't confidently answer that question" and instead let their systems get away with low confidence answers.
Summary: Be skeptical where you need AI most
Most users scale their trust in AI with their own confidence about a topic: I feel good about this answer, so it probably is good. The correct calibration is the opposite. The less familiar the territory, the more likely both that the model is extrapolating and that the user won't notice if it's wrong.
Errors you can catch are on topics you can independently verify. The errors you can't catch are on topics you can't verify — the same topics where hallucinations cluster most densely. The two distributions don't just coincide. They reinforce each other.
So essentially, when using AI, we have to revert our instinctive reaction to "trust the expert" - the one who presents facts about a topic less known to us with confidence. In case of AI, it typically is mirroring our own confidence levels.
How to responsibly use AI
There are two different takeaways from the fact that hallucinations are increasing in less-documented areas - they differ by use case.
For developers of retrieval systems that solve specific problems a key rule must be to always err towards caution when stating answers. When vector data suggests low similarity, drop the answer or state explicitly that the confidence iin its correctness s low.
For most of us whojust use Large Language Models in their publicly available form: we need to retrain our brain in how we treat answers provided by those models. And for companies whose employees use AI, be it in an officially approved way or just quietly, training is the essential answer to the problem.