AI Hallucinations: A Confident Confabulator

About this post

We examine why AI hallucinations concentrate in topics where users are least equipped to notice them - and why this structural pattern makes them a usability problem, not just a technical one.

Key takeaways

Die Halluzinationsquote ist invers mit der Verfügbarkeit von Trainingsdaten korreliert. Themen mit wenig Inhalt produzieren mehr Fehler und werden von Nutzern weniger oft entdeckt. Je weniger wir wissen, desto weniger sollten wir KI-Antworten vertrauen.

The original ideas, structure, and much of the language are human-created, but AI was used to develop, enrich, or rework portions of the content — for example, researching sources, rewriting sections for clarity, or expanding on arguments.

Ask an Artificial Intelligence model about the French Revolution, and you get a competent summary. Ask about a minor 18th-century poet, and you’ll still get a competent-sounding summary – except this one might be partly invented, with fabricated quotes and a biography stitched together from similar figures. Both answers are provided with the same polished authority of an expert in the field. One is real, one is mostly fiction.

This isn’t a random failure; AI hallucinations are not evenly distributed across topics. They cluster, systematically and predictably, in exactly those areas where users know less and are thus least able to detect them. This makes them one of the biggest problems of AI.

So in short: the more thoroughly a subject has been written about and made available to the AI model, the more reliably it can talk about it. The further a query drifts into specialist territory, the more the output shifts from recall to invention. This is not a calibration problem that training alone will solve – it’s intrinsic to how these systems work. The pattern shows up consistently wherever it’s been measured and is confirmed by many studies.

The failures will be in cases where it’s harder for a reader to notice – because they are more obscure.

Emily M. Bender, University of Washington, 2023

The table below provides some data on hallucination levels across multiple topics. The results show that hallucinations are negatively correlated with domain knowledge – the more information is available, the less AI models confabulate.

Domain / Task	Hallucination Rate	Source
Short-document summarization (top models)	0.7–1.5%	Vectara HHEM Leaderboard
Citations on well-studied medical topic (major depression)	~6%	Deakin University, 2025
Citations on less-studied medical topic (body dysmorphic disorder)	~29%	Deakin University, 2025
Real-world conversational benchmark	31.4%	AuthenHallu, arXiv:2510.10539, 2025
Purpose-built legal AI tools (Lexis+, Westlaw)	17–34%	Stanford RegLab / HAI, 2024
General LLMs on specific legal queries	58–88%	Stanford “Hallucinating Law”, 2024
Requesting information on non-existent libraries	up to 99%	arXiv:2509.22202, 2025

Sources: Vectara Hallucination Leaderboard (github.com/vectara/hallucination-leaderboard); Deakin University, ChatGPT citation accuracy in mental health literature reviews, 2025; Stanford HAI, “Hallucinating Law” and “Hallucination-Free?”, 2024; AuthenHallu benchmark, arXiv:2510.10539, October 2025; Library Hallucinations in LLMs, arXiv:2509.22202, 2025.

The core problem: How AI stores and retrieves

How Language Models see text

Text never enters or leaves a language model as text. It gets broken into tokens – subword fragments, each mapped to an integer ID – and each token is then converted into a high-dimensional vector, typically 1,000 to 10,000 numbers long. That vector is everything the model “knows”. Training adjusts billions of parameters so that tokens appearing in similar contexts end up with mathematically similar vectors, and so the model can predict which vector is most likely to come next.

At no point does the system check anything against truth. It checks proximity. The sentences “penicillin was discovered by Alexander Fleming” and “penicillin was discovered by Alexander Flemming” are nearly identical as vectors – the model outputs whichever pattern the training data made statistically more likely, not whichever is factually correct. Where training data is thin, the nearest statistical neighbor is often too far away to represent a fact, and the model presents the wrong answer dressed in the right shape – with confidence.

The mechanism behind this relationship is not mysterious – it’s directly baked into how language models process language in the first place. There is no actual recognition of content or conscious creation of text, just a very refined vector comparison that finds the nearest possible match and the most likely continuation. If there are many finds on a subject, “near” means that it is almost always correct. If there is limited source material, “near” can be too far away. Once we understand this fundamental concept of how a query actually moves through the system, the accumulation of hallucinations in less-covered subjects is a logical consequence.

This all applies to us humans too. Each of us has areas of expertise where we can confidently state facts and rarely make mistakes and others where our knowledge is limited.

The key difference is the confidence with which uncertainty is presented. For AI systems, confidence is baked into their generation logic, as it is not connected to the validity of the content they are presenting. They sound authoritative and certain when making things up.

This confidence mismatch compounds the problem. Humans use linguistic confidence as a heuristic for reliability – nervous, hedging speech signals uncertainty; fluent assertion signals knowledge. Language models break this cue. They speak with uniform fluency whether they’re on solid ground or free-associating from fragments, and the signal humans rely on to gauge speaker reliability is, for practical purposes, absent from their output. And most LLM companies decided that users don’t like a “sorry, I can’t confidently answer that question” and instead let their systems get away with low confidence answers.

Summary: Be skeptical where you need AI most

Most users scale their trust in AI with their own confidence about a topic: I feel good about this answer, so it probably is good. The correct calibration is the opposite. The less familiar the territory, the more likely both that the model is extrapolating and that the user won’t notice if it’s wrong.

Errors you can catch are on topics you can independently verify. The errors you can’t catch are on topics you can’t verify – the same topics where hallucinations cluster most densely. The two distributions don’t just coincide. They reinforce each other.

So essentially, when using AI, we have to reverse our instinctive reaction to “trust the expert” – the one who presents facts about a topic less known to us with confidence. In case of AI, it typically is mirroring our own confidence levels.

Referenzen und weitere Quellen

Research and Benchmarks

AuthenHallu benchmark. (2025). arXiv preprint arXiv:2510.10539.

Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024). Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1).

Library hallucinations in LLMs: Risk analysis grounded in developer queries. (2025). arXiv preprint arXiv:2509.22202.

Linardon, J., Jarman, H. K., McClure, Z., Anderson, C., Liu, C., & Messer, M. (2025). Influence of topic familiarity and prompt specificity on citation fabrication in mental health research using large language models: Experimental study. JMIR Mental Health, 12, e80371. DOI: 10.2196/80371.

Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2024). Hallucination-free? Assessing the reliability of leading AI legal research tools. Stanford HAI.

Vectara. Hallucination Leaderboard. github.com/vectara/hallucination-leaderboard.

Commentary

O'Brien, M. (2023). Chatbots sometimes make things up. Is AI's hallucination problem fixable? Associated Press.

Der neue Baron von Münchhausen