{"id":225031,"date":"2026-04-27T14:33:47","date_gmt":"2026-04-27T14:33:47","guid":{"rendered":"https:\/\/www.9senses.ai\/?p=225031"},"modified":"2026-07-17T07:51:23","modified_gmt":"2026-07-17T07:51:23","slug":"the-ai-feedback-lottery","status":"publish","type":"post","link":"https:\/\/www.9senses.ai\/de\/the-ai-feedback-lottery\/","title":{"rendered":"The AI Feedback Lottery"},"content":{"rendered":"
<\/div>\n\n
\n
\n
\n

Du hast nicht ausreichende Berechtigungen, um auf diesen Inhalt zuzugreifen.<\/p>\n<\/div><\/div>\n\n

The AI Feedback Lottery<\/h1><\/div><\/div>\n\n

Large Language Models judge your work differently every time you ask. That is not just a quirk of the technology \u2014 it is a fundamental challenge to how we create, validate, and trust ideas.<\/p>\n<\/div><\/div>\n\n

Imagine you have spent weeks on a business plan. You show it to an AI assistant, and it comes back glowing: \"This is a compelling, well-structured concept - you are ready to move forward.\" Energized, you close the chat window and get to work. A few days later, curiosity gets the better of you. You open a fresh conversation, paste in the exact same plan with the exact same prompt, and wait. The verdict this time? \"There are significant structural weaknesses here. The market analysis is underdeveloped, the financial projections rest on untested assumptions, and the value proposition needs a substantial rethink.\"<\/span><\/p>\n

Same document. Same question. Two completely different realities.<\/span><\/p>\n

This is not a bug. It is one of the most underappreciated structural truths about working with Large Language Models - and it carries real consequences for anyone using AI as a creative partner, a sounding board, or a validator of their ideas.<\/span><\/p>\n

<\/span><\/p>\n<\/div><\/div>\n\n

LLMs can't give you a consistent opinion<\/h1><\/div><\/div>\n<\/div>\n\n
\n
Vector map \u2014 small<\/div>
Vector map \u2014 small<\/div>[ninesenses_vectormap #3]<\/code>
Max height: 230px. Preview on the live page.<\/div><\/div><\/div><\/div><\/code>
Max height: 230px. Preview on the live page.<\/div><\/div><\/div><\/div><\/div><\/div>\n\n

About this post<\/h3><\/div><\/div>\n\n

Du hast nicht ausreichende Berechtigungen, um auf diesen Inhalt zuzugreifen.<\/p>\n<\/div><\/div>\n\n

Key takeaways<\/h3><\/div><\/div>\n\n

Du hast nicht ausreichende Berechtigungen, um auf diesen Inhalt zuzugreifen.<\/p>\n<\/div><\/div>\n\n

Du hast nicht ausreichende Berechtigungen, um auf diesen Inhalt zuzugreifen.<\/p>\n<\/div><\/div>\n<\/div>\n<\/div>\n<\/div>\n\n

\n
\n
\n

A Large Language Model does not hold stable opinions the way a human collaborator does. Each response is generated anew from probabilities shaped by the current conversation context. Clear the session, and the model approaches your work from scratch again - which means the same prompt can produce materially different judgments across sessions, especially for subjective or creative work.<\/span><\/p>\n<\/div><\/div>\n<\/div>\n<\/div>\n\n

\n
\n

Session One<\/h3>\n

\"This is an excellent foundation. The core idea is strong, the structure is sound, and the execution is ready for prime time. I would move forward with confidence.\"<\/p>\n<\/div><\/div>\n<\/div>\n\n

\n

Session Two<\/h3>\n

\"This has potential, but the concept has a number of flaws that would need to be addressed. I would recommend a substantial rework before proceeding.\"<\/p>\n<\/div><\/div>\n<\/div>\n<\/div>\n\n

\n
\n

Both responses come from the same model. Both are coherent, well-reasoned, and written with conviction. Neither is lying. But they cannot both be right - and yet the model will defend either one with equal fluency if you ask it to. Technically, the model is not retrieving a fixed evaluation stored somewhere internally. Even with the identical prompt, each generation involves probabilistic token sampling, where multiple plausible continuations compete, and small statistical differences cascade into different overall responses. In creative domains - where there is no single correct answer - those branching probabilities can lead the model toward enthusiasm in one session and scepticism in another.<\/span><\/span><\/p>\n<\/div><\/div>\n<\/div>\n<\/div>\n\n

\n
\n

J.K. Rowling and the luck of the draw<\/h1><\/div><\/div>\n<\/div>\n\n
<\/div>\n<\/div>\n\n
\n
\n

When J.K. Rowling finished the first Harry Potter manuscript, she received twelve verdicts from professional editors at different publishers, and every single one of them was a rejection. The story that would become one of the most successful children's book series of the past decades was, by the consensus of expert opinion at the time, not worth publishing.<\/span><\/p>\n

The editors were making genuine, experienced judgments - but those judgments were shaped by the mood of the reader, the publisher's current preferences, the editor's personal sensibility that day, and a thousand other unknown variables. Creative evaluation has always been, at its core, a probabilistic process. AI has simply made that probabilistic chaos instantaneous, frictionless, and disconnected from reality.<\/span><\/p>\n<\/div><\/div>\n<\/div>\n\n

\n

What changes with AI is not inconsistency itself, but our tendency to mistake the confident, well-structured prose of an AI response for a stable, reliable, and authoritative verdict. The deeper issue is that all creative judgment is inherently inconsistent: human taste shifts, markets fluctuate, editorial fashions change. Yet AI\u2019s extraordinary fluency conceals this instability. A response generated by a Large Language Model carries the tone of a considered opinion because it has been trained on text written by humans expressing considered opinions. The surface markers of certainty are all present. The underlying stability is not.<\/p>\n

Rowling\u2019s rejection letters, at least, arrived with the implicit understanding that they reflected human judgments - fallible, contextual, one among many. An AI response, by contrast, arrives wrapped in the aesthetic of authority, when in reality it is nothing more than a stochastically generated synthesis of past human opinion as captured on the internet.<\/p>\n<\/div><\/div>\n<\/div>\n<\/div>\n\n

\n
\n

After those rejections, J.K. Rowling sent out her manuscript another thirteenth time, and it opened the doors to the successful launch of Harry Potter and the Philosopher's Stone in 1997.<\/span><\/p>\n<\/div><\/div>\n<\/div>\n<\/div>\n\n

\n
\n

Three ways to deal with LLM feedback<\/h1><\/div><\/div>\n<\/div>\n\n
<\/div>\n<\/div>\n\n
\n
\n

When creators come across an inconsistency in AI feedback, there are three ways to respond, and only one leads to an efficient evaluation process. It requires the skill to know what the tool is - and what it is not.<\/p>\n<\/div><\/div>\n<\/div>\n<\/div>\n\n

\n
\n

The Deflated Creator<\/h3><\/div><\/div>\n\n

A self-critical person may respond to this kind of AI feedback discrepancy with discouragement. If the AI loved your work this morning and dismissed it this afternoon, what does that say about the work? For many people, especially those already prone to self-doubt, a single harsh AI verdict can be enough to shelve an idea entirely. The cruelty is that the harsh verdict may simply have been a statistical outlier: the equivalent of one editor at one publisher having a bad day. But without the context that it was only one draw from a probabilistic deck, it reads as definitive judgment while also devaluing the earlier praise.<\/p>\n<\/div><\/div>\n<\/div>\n\n

\n

The Overconfident Creator<\/h3><\/div><\/div>\n\n

The second response is the mirror image: unfounded confidence. A confident creator shops sessions until they get the enthusiastic verdict they were hoping for, takes a screenshot, and proceeds as though the AI has validated their work. This is a perfectly natural psychological response to an uncertain situation. But it is dangerous, because the encouragement carries no more weight than the discouragement. It is simply a lucky draw, not an informed assessment. The creator moves forward with conviction built on sand.<\/span><\/p>\n<\/div><\/div>\n<\/div>\n\n

\n

The Sophisticated Creator<\/h3><\/div><\/div>\n\n

The third response is the only genuinely useful one: treating the inconsistency as information in itself. A creator who asks the same AI the same question multiple times - or who deliberately varies the framing to stress-test an idea - is not gaming the system. They are doing something close to what a good editor or creative director does when they seek multiple opinions before committing to a direction. The variance in the responses tells them something real: where the idea is genuinely strong (consistent praise across sessions), and where it is genuinely fragile (inconsistent or conflicting feedback).<\/span><\/p>\n<\/div><\/div>\n<\/div>\n<\/div>\n\n

\n
\n

How to use AI as a creative partner?<\/h1><\/div><\/div>\n<\/div>\n<\/div>\n\n
\n
\n

None of this means AI feedback is worthless. A single AI session can surface blind spots, identify unclear passages, suggest alternatives you had not considered, and push you toward a more refined version of your idea.<\/p>\n

A good way to go about this is to use the statistical distribution to your advantage. If you ask the LLM the same question three times with the same prompt, you will likely get a solid distribution of praise and criticism. And if you want, you can additionally outsource the task to find the similarities and differences in these evaluations to your LLM.<\/p>\n<\/div><\/div>\n<\/div>\n\n

\n

One challenge in this context is \u201cpersistent memory\u201d, where the Large Language Model stores the content of previous conversations. If this is included in the second and third evaluations, it dilutes the effect of giving you a fresh perspective and will continue to justify previous statements. Turn it off!<\/p>\n

One caveat: The more novel and outside-of-the-box your idea is, the less a LLM will be able to provide you with content feedback. It is built on past knowledge and patterns, and not well equipped to detect valuable novelty that is beyond extending what already exists. Thus, you can still spell-check and check for completeness, but don\u2019t expect a solid evaluation of the substantive content.<\/p>\n<\/div><\/div>\n<\/div>\n<\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"

Gro\u00dfe Sprachmodelle bewerten Dinge jedes Mal anders, wenn wir sie um Feedback bitten. Diese KI-Eigenart wird zur grundlegenden Herausforderung, wenn wir Ideen entwickeln und \u00fcberpr\u00fcfen wollen.<\/p>","protected":false},"author":1,"featured_media":225034,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[47,50],"tags":[],"class_list":["post-225031","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-homepage"],"acf":{"tag_line":"When AI Answers differ each time","about":"This post explores a structural characteristic of Large Language Models and its practical implications for creators, entrepreneurs, and anyone using AI as a thinking partner.","tldr":"If you want to use AI for feedback, do it with caution and don't take everything it says for granted. And do multiple runs as if you would ask multiple people.","ai_support":"The original ideas, structure, and much of the language are human-created, but AI was used to develop, enrich, or rework portions of the content \u2014 for example, researching sources, rewriting sections for clarity, or expanding on arguments - please click for more information"},"_links":{"self":[{"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/posts\/225031","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/comments?post=225031"}],"version-history":[{"count":20,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/posts\/225031\/revisions"}],"predecessor-version":[{"id":226720,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/posts\/225031\/revisions\/226720"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/media\/225034"}],"wp:attachment":[{"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/media?parent=225031"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/categories?post=225031"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.9senses.ai\/de\/wp-json\/wp\/v2\/tags?post=225031"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}