{"id":227597,"date":"2026-04-20T08:35:14","date_gmt":"2026-04-20T08:35:14","guid":{"rendered":"https:\/\/www.9senses.ai\/ns-lab-a-confident-confabulator\/"},"modified":"2026-07-31T06:45:25","modified_gmt":"2026-07-31T06:45:25","slug":"a-confident-confabulator","status":"publish","type":"post","link":"https:\/\/www.9senses.ai\/de\/a-confident-confabulator\/","title":{"rendered":"Der neue Baron von M\u00fcnchhausen"},"content":{"rendered":"<p>Ask an Artificial Intelligence model about the French Revolution, and you get a competent summary. Ask about a minor 18th-century poet, and you&#8217;ll still get a competent-sounding summary &#8211; except this one might be partly invented, with fabricated quotes and a biography stitched together from similar figures. Both answers are provided with the same polished authority of an expert in the field. One is real, one is mostly fiction.<\/p>\n<p>This isn&#8217;t a random failure; AI hallucinations are not evenly distributed across topics. They cluster, systematically and predictably, in exactly those areas where users know less and are thus least able to detect them. This makes them one of the biggest problems of AI.<\/p>\n<p>So in short: the more thoroughly a subject has been written about and made available to the AI model, the more reliably it can talk about it. The further a query drifts into specialist territory, the more the output shifts from recall to invention. This is not a calibration problem that training alone will solve &#8211; it&#8217;s intrinsic to how these systems work. The pattern shows up consistently wherever it&#8217;s been measured and is confirmed by many studies.<\/p>\n<blockquote class=\"ns-obj ns-obj--quote\"><p>The failures will be in cases where it&#8217;s harder for a reader to notice &#8211; because they are more obscure.<\/p><cite class=\"ns-quote-cite\">Emily M. Bender, University of Washington, 2023<\/cite><\/blockquote>\n<p>The table below provides some data on hallucination levels across multiple topics. The results show that hallucinations are negatively correlated with domain knowledge &#8211; the more information is available, the less AI models confabulate.<\/p>\n<figure class=\"ns-obj ns-obj--tablewrap ns-obj--full ns-obj--fmt-normal\">\n<table class=\"data\">\n<thead>\n<tr>\n<th>Domain \/ Task<\/th>\n<th>Hallucination Rate<\/th>\n<th>Source<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Short-document summarization (top models)<\/td>\n<td>0.7\u20131.5%<\/td>\n<td>Vectara HHEM Leaderboard<\/td>\n<\/tr>\n<tr>\n<td>Citations on well-studied medical topic (major depression)<\/td>\n<td>~6%<\/td>\n<td>Deakin University, 2025<\/td>\n<\/tr>\n<tr>\n<td>Citations on less-studied medical topic (body dysmorphic disorder)<\/td>\n<td>~29%<\/td>\n<td>Deakin University, 2025<\/td>\n<\/tr>\n<tr>\n<td>Real-world conversational benchmark<\/td>\n<td>31.4%<\/td>\n<td>AuthenHallu, arXiv:2510.10539, 2025<\/td>\n<\/tr>\n<tr>\n<td>Purpose-built legal AI tools (Lexis+, Westlaw)<\/td>\n<td>17\u201334%<\/td>\n<td>Stanford RegLab \/ HAI, 2024<\/td>\n<\/tr>\n<tr>\n<td>General LLMs on specific legal queries<\/td>\n<td>58\u201388%<\/td>\n<td>Stanford &#8220;Hallucinating Law&#8221;, 2024<\/td>\n<\/tr>\n<tr>\n<td>Requesting information on non-existent libraries<\/td>\n<td>up to 99%<\/td>\n<td>arXiv:2509.22202, 2025<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span class=\"table-sources\">Sources: Vectara Hallucination Leaderboard (github.com\/vectara\/hallucination-leaderboard); Deakin University, ChatGPT citation accuracy in mental health literature reviews, 2025; Stanford HAI, &#8220;Hallucinating Law&#8221; and &#8220;Hallucination-Free?&#8221;, 2024; AuthenHallu benchmark, arXiv:2510.10539, October 2025; Library Hallucinations in LLMs, arXiv:2509.22202, 2025.<\/span><\/figure>\n<h2>The core problem: How AI stores and retrieves<\/h2>\n<aside class=\"ns-obj ns-obj--aside ns-obj--right ns-obj--c1 ns-obj--fmt-normal\" style=\"width:55%\"><div class=\"ns-obj-body\"><h3>How Language Models see text<\/h3>\n<p>Text never enters or leaves a language model as text. It gets broken into tokens &#8211; subword fragments, each mapped to an integer ID &#8211; and each token is then converted into a high-dimensional vector, typically 1,000 to 10,000 numbers long. That vector is everything the model &#8220;knows&#8221;. Training adjusts billions of parameters so that tokens appearing in similar contexts end up with mathematically similar vectors, and so the model can predict which vector is most likely to come next.<\/p>\n<p>At no point does the system check anything against truth. It checks proximity. The sentences &#8220;penicillin was discovered by Alexander Fleming&#8221; and &#8220;penicillin was discovered by Alexander Flemming&#8221; are nearly identical as vectors &#8211; the model outputs whichever pattern the training data made statistically more likely, not whichever is factually correct. Where training data is thin, the nearest statistical neighbor is often too far away to represent a fact, and the model presents the wrong answer dressed in the right shape &#8211; with confidence.<\/p><\/div><\/aside>\n<p>The mechanism behind this relationship is not mysterious &#8211; it&#8217;s directly baked into how language models process language in the first place. There is no actual recognition of content or conscious creation of text, just a very refined vector comparison that finds the nearest possible match and the most likely continuation. If there are many finds on a subject, &#8220;near&#8221; means that it is almost always correct. If there is limited source material, &#8220;near&#8221; can be too far away. Once we understand this fundamental concept of how a query actually moves through the system, the accumulation of hallucinations in less-covered subjects is a logical consequence.<\/p>\n<p>This all applies to us humans too. Each of us has areas of expertise where we can confidently state facts and rarely make mistakes and others where our knowledge is limited.<\/p>\n<p>The key difference is the confidence with which uncertainty is presented. For AI systems, confidence is baked into their generation logic, as it is not connected to the validity of the content they are presenting. They sound authoritative and certain when making things up.<\/p>\n<p>This confidence mismatch compounds the problem. Humans use linguistic confidence as a heuristic for reliability &#8211; nervous, hedging speech signals uncertainty; fluent assertion signals knowledge. Language models break this cue. They speak with uniform fluency whether they&#8217;re on solid ground or free-associating from fragments, and the signal humans rely on to gauge speaker reliability is, for practical purposes, absent from their output. And most LLM companies decided that users don&#8217;t like a &#8220;sorry, I can&#8217;t confidently answer that question&#8221; and instead let their systems get away with low confidence answers.<\/p>\n<h2>Summary: Be skeptical where you need AI most<\/h2>\n<aside class=\"ns-obj ns-obj--aside ns-obj--right ns-obj--c1 ns-obj--fmt-inverted\" style=\"width:55%\"><div class=\"ns-obj-body\"><h3>How to responsibly use AI<\/h3>\n<p>There are two different takeaways from the fact that hallucinations are increasing in less-documented areas &#8211; they differ by use case.<\/p>\n<p>For developers of retrieval systems that solve specific problems a key rule must be to always err towards caution when stating answers. When vector data suggests low similarity, drop the answer or state explicitly that the confidence in its correctness is low.<\/p>\n<p>For most of us who just use Large Language Models in their publicly available form: we need to retrain our brain in how we treat answers provided by those models. And for companies whose employees use AI, be it in an officially approved way or just quietly, training is the essential answer to the problem.<\/p><\/div><\/aside>\n<p>Most users scale their trust in AI <em>with<\/em> their own confidence about a topic: I feel good about this answer, so it probably is good. The correct calibration is the opposite. The less familiar the territory, the more likely both that the model is extrapolating and that the user won&#8217;t notice if it&#8217;s wrong.<\/p>\n<p>Errors you can catch are on topics you can independently verify. The errors you can&#8217;t catch are on topics you can&#8217;t verify &#8211; the same topics where hallucinations cluster most densely. The two distributions don&#8217;t just coincide. They reinforce each other.<\/p>\n<p>So essentially, when using AI, we have to reverse our instinctive reaction to &#8220;trust the expert&#8221; &#8211; the one who presents facts about a topic less known to us with confidence. In case of AI, it typically is mirroring our own confidence levels.<\/p>","protected":false},"excerpt":{"rendered":"<p>AI hallucinations aren&#8217;t random &#8211; they cluster, systematically and predictably, in the topics you cannot independently verify.<\/p>","protected":false},"author":20,"featured_media":225005,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"ns_references":"<h3>Research and Benchmarks<\/h3>\r\n<p><em>AuthenHallu benchmark. (2025). arXiv preprint arXiv:2510.10539.<\/em><\/p>\r\n<p><em>Dahl, M., Magesh, V., Suzgun, M., &amp; Ho, D. E. (2024). Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1).<\/em><\/p>\r\n<p><em>Library hallucinations in LLMs: Risk analysis grounded in developer queries. (2025). arXiv preprint arXiv:2509.22202.<\/em><\/p>\r\n<p><em>Linardon, J., Jarman, H. K., McClure, Z., Anderson, C., Liu, C., &amp; Messer, M. (2025). Influence of topic familiarity and prompt specificity on citation fabrication in mental health research using large language models: Experimental study. JMIR Mental Health, 12, e80371. DOI: 10.2196\/80371.<\/em><\/p>\r\n<p><em>Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., &amp; Ho, D. E. (2024). Hallucination-free? Assessing the reliability of leading AI legal research tools. Stanford HAI.<\/em><\/p>\r\n<p><em>Vectara. Hallucination Leaderboard. github.com\/vectara\/hallucination-leaderboard.<\/em><\/p>\r\n\r\n<h3>Commentary<\/h3>\r\n<p><em>O'Brien, M. (2023). Chatbots sometimes make things up. Is AI's hallucination problem fixable? Associated Press.<\/em><\/p>","ns_references_title":"References and further reading","footnotes":""},"categories":[47,50],"tags":[],"class_list":["post-227597","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-homepage"],"acf":{"tag_line":"Why generative AI fails exactly where you don't notice","about":"We examine why AI hallucinations concentrate in topics where users are least equipped to notice them - and why this structural pattern makes them a usability problem, not just a technical one.","tldr":"Hallucination rates track inversely with training data density. Topics with little data produce both higher error rates and lower user detection. Skepticism needs to scale inversely with your own expertise on a subject.","ai_support":"The original ideas, structure, and much of the language are human-created, but AI was used to develop, enrich, or rework portions of the content \u2014 for example, researching sources, rewriting sections for clarity, or expanding on arguments."},"_links":{"self":[{"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/posts\/227597","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/users\/20"}],"replies":[{"embeddable":true,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/comments?post=227597"}],"version-history":[{"count":7,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/posts\/227597\/revisions"}],"predecessor-version":[{"id":227804,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/posts\/227597\/revisions\/227804"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/media\/225005"}],"wp:attachment":[{"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/media?parent=227597"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/categories?post=227597"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/tags?post=227597"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}