Führt uns KI in die Apokalypse?

Führt uns KI zum Weltuntergang?

Namhafte KI-Experten sehen eine realistische Chance, dass KI unsere Zivilisation zerstören kann. Wir haben die einfache Antwort: Es ist nicht die KI, wir sind's.

"Ich sehe eine Chance von 25%, dass sich die Dinge sehr sehr negativ entwickeln". Das ist nicht die Stimme eines KI-Gegners, dieses Zitat stammt von Dario Amodei, dem CEO von Anthropic, der Firma, die Claude entwickelt hat. Diese Wahrscheinlichkeit sieht er dafür, dass KI unserer Zivilisation ein Ende setzen könnte. Natürlich verwettet er die anderen 75% darauf, dass sich die Welt dank KI positiv entwickelt. Dennoch stellt sich die Frage, wieso jemand, der so nahe an der Technologie dran ist, der so viel durch sie verdienen kann, eine solche Möglichkeit überhaupt in Erwägung zieht?

Alleine ist er mit dieser Einschätzung nicht; nicht wenige Menschen mit tiefgreifendem Wissen haben ähnliche Befürchtungen geäußert. Aber fürchten sie sich wirklich davor, dass die Maschine eines Tages beschließt, dass wir Menschen überflüssig sind und uns abschafft? Das ist eher unwahrscheinlich, auch weil man die KI einfach dadurch stoppen kann, dass man den Stecker ihrer energiefressenden Rechenzentren zieht. Aber es gibt ein anderes und nicht zu unterschätzendes Risiko. Es fängt mit dem Verständnis dessen an, was KI ist und was sie eben nicht ist, und was wir uns unter KI vorstellen. Bevor wir also in Panik verfallen oder KI als harmlos beurteilen, ist eine etwas tiefergehende Analyse dringend notwendig.

Der wichtigste Punkt zuerst, den die meisten Menschen nicht im Blick haben: Künstliche Intelligenz hat keine Ahnung davon, wovon sie spricht. Selbst der beste Algorithmus baut auf der Übereinstimmung von Vektoren auf, auch wenn das Modell uns "versteht" und mit uns "spricht". Verstehen und Sprechen bedeuteten eben gerade nicht, dass es unsere Welt auf eine uns vergleichbare Art und Weise begreift. Mustererkennung bedeutet eben nicht, dass der Sinn der Inhalte für die Maschine klar ist. Wenn uns die KI sagt, dass ein Medikament keine gefährlichen Nebenwirkungen hat, dann baut das nicht auf medizinischem Verständnis auf, sondern darauf, wie eine richtige Antwort im Normalfall früher ausgesehen hat.

Thema

Wir befassen uns mit Aussagen namhafter KI-Experten, die katastrophale Auswirkungen auf die Menschheit befürchten. Eine Analyse der Funktion und unserer Nutzung von KI soll dazu ein wenig Klarheit schaffen.

Zusammenfassung

Nein, KI wird unsere Welt nicht zugrunde richten. Wenn wir allerdings KI auf eine Art und Weise nutzen, die ihre Fähigkeiten übersteigt, geben wir ihr die Entscheidungsgewalt über Dinge, die immer von Menschen entschieden werden sollten.

Dieser Text wurde von einem Menschen verfasst und einem KI-System zur abschließenden Überprüfung vorgelegt, beispielsweise zur Überprüfung der Grammatik, auf Tippfehler oder auf logische Konsistenz.

"Das Risiko unserer Ausrottung durch KI sollte genau so viel Priorität haben wie andere gesellschaftsweite Risiken wie Pandemien und Atomkriege."

Center for AI Safety, 2023

Im weiteren ist das Innovationspotential von KI konzeptbedingt beschränkt. Sie kann beeindruckende Dinge leisten, die sich wie Kreativität anfühlen, sie kann sogar Lieder komponieren und Gedichte schreiben. Aber sie kann das nur, weil sie eine große Menge an Lyrik und Musik gesehen hat, und weiß, was bei den Menschen angekommen ist. Dies gibt ihr genügend Information zur Interpolation. In der Wissenschaft verfügt sie über das verfügbare Wissen der Menschheit zu Physik, Chemie und Biologie. Als zum Beispiel AlphaFold Strukturen vorhergesagt hat, die vorher kein Biochemiker gefunden hatte, oder als generative KI neue chemische Substanzen vorgeschlagen, die bislang noch nie synthetisiert wurden, ist dies das Ergebnis der Anwendung bekannter Regel und deren Extrapolation, und keine bahnbrechende Erfindung.

Und hier sitzt die Grenze. Entdeckungen durch KI loten den bekannten Wissensraum aus - durch Rekombination und Interpolation, und erweitern diesen entlang bekannter Dimensionen. Was KI genuin nicht kann ist, in einem konsistenten Rahmen Fehler zu erkennen oder Inkonsistenzen in den darunterliegenden Annahmen aufzudecken - geschweige denn neue Konzepte oder Annahmen zu entwickeln.

Think about the discovery of penicillin - Alexander Fleming noticing something strange in a petri dish and following a hunch that nothing in prior science predicted what he saw. Or the germ theory of disease, which required someone to reject the dominant concept entirely and propose something that looked absurd at the time. These discoveries were specific ruptures from patterns, no extensions. That kind of leap - seeing past the boundaries of established knowledge into genuinely unmapped territory - remains a distinctly human capability that AI is ill-equipped to make.

So why should we be scared?

The pattern matching limit

There are objections on our assertion that "mere pattern matching" is a true limitation for AI and it all relies on the definition of intelligence. If we define intelligence narrowly as representation of knowledge or problem-solving ability in a cleanly known and documented domain, AI outperforms any human expert.

But that this is the essence of intelligence is not true. Human intelligence comprises the contextualization of content into a world image formed over years, which - correct or not - guides decision. AI doesn't have that construct internally and will not have it as long as it's built based on vectors mapping the past. 

If AI has these fundamental limitations, why lose sleep over it? Because the danger was never really about AI deciding humanity is a problem and staging a coup; that's a cool movie plot, but not what is likely to happen. The real risk from AI is far more mundane and far more plausible. The real risk is us.

Specifically the risk is that we - individually and collectively - begin to systematically replace our own judgment with AI outputs, without adequately accounting for what AI cannot do. That process is already quietly underway across thousands of decision-making contexts. A few examples where that could go wrong: a financial institution using an AI risk model trained on an economic environment that no longer exists. Or a hospital using an AI diagnostic tool so reliably that doctors gradually stop scrutinizing the edge cases the model has never really seen. Or a government using AI-assisted policy analysis built on data that systematically underrepresents the communities most affected by its actions. None of these scenarios requires an AI to "go rogue" - they just require humans to gradually cede their judgment to a system that, by design, does not exercise human judgment.

And by doing so, we further erode the capabilities of decision-making, both by losing the ability to apply that judgment ourselves, and by making prior, potentially unsound, AI decisions the baseline for future decision-making without sufficient re-evaluation.

The problem is giving AI too much power

The bottom line is that any existential threat from AI doesn't come from it suddenly developing malicious intent, but instead  from us misinterpreting its capabilities so profoundly that we will stop providing the essential human ingredient: genuine judgment and accountability. As AI and human judgment are qualitatively different, they fail in different ways, and for different reasons. A decision-making system that relies entirely on AI collapses that diversity. When the AI's failure mode triggers - novel situations, distributional shift, pattern-matching on a flawed historical baseline - there's no human backstop left.

This is the mechanism by which AI could contribute to catastrophic outcomes. Not through superintelligence, but through the quiet erosion of the human judgment that was always supposed to remain in the loop. And by an overexpansion of rights we grant AI. So ultimately, the extinction-level risk isn't really about AI. It's about us, reflected back through AI.

So the question "could AI lead to human extinction?" is really asking: "could we sleepwalk into catastrophe by systematically mistaking pattern-matching for understanding, and outsourcing our judgment to a very sophisticated mirror of our own past?" That's a question worth taking seriously.

Who decides when AI pulls the trigger?

All of this has publicly surfaced recently. In July 2025, Anthropic signed a $200 million contract with the US Department of Defense, becoming the first AI company to deploy its Claude models on classified government networks. The partnership looked like a pragmatic compromise: Claude would support intelligence analysis, operational planning, and cyber operations, but under two explicit restrictions. Anthropic would not allow it to be used for mass domestic surveillance of American citizens, or to power fully autonomous weapons systems - weapons that, once activated, select and engage targets without any human in the loop.

The Pentagon pushed back. Defense Secretary Pete Hegseth and the DoD characterized those limits as unduly restrictive, insisting that responsible AI should encompass "all lawful purposes" by the US military. Anthropic refused to remove the restrictions. The DoD responded by designating Anthropic a supply chain risk - a label previously reserved for foreign adversaries - requiring defense vendors and contractors to certify they don't use Anthropic's models in their Pentagon work.

It's worth thinking about that twice: fully autonomous weapons. AI already assists in military decision-making and Anthropic didn't object to that. The debate is about whether AI should be permitted to make the final lethal decision, with no human required to authorize it.

Anthropic's position was that deploying unreliable AI in autonomous weapons would endanger American soldiers, not protect them - a technical argument as much as an ethical one. Based on this, the administration blacklisted the AI company most deeply integrated into its own classified networks, even as US strikes in Iran reportedly used Anthropic's technology hours after the ban was announced.

What makes this episode so significant is what the pressure on Anthropic actually was: not to make Claude smarter or more capable, but to remove the human judgment layer - to strip out the guardrails that kept humans accountable for lethal decisions. That is a near-perfect illustration of the risk in action. The danger isn't that AI will autonomously decide to wage war. It's that humans will deliberately choose to remove themselves from the decision - and call it efficiency.

It is still our call

The risks described above are not baked into AI technology. They arise from our choices - about how we deploy AI, what decisions we allow it to make autonomously, and what kinds of human oversight we deliberately preserve.

AI can be an extraordinary tool for augmenting human judgment, not replacing it. Used well, it extends our reach, surfaces patterns we might miss, and handles complexity at a scale no human team could match. The institutions that will navigate the AI era well are the ones that maintain a clear-eyed understanding of what AI can and cannot do - and that deliberately protect the human ingenuity and accountability no model can fully replicate.

Why making war decisions with AI is a particularly bad idea

The idea of using AI in war deserves particular scrutiny, particularly when it comes to allowing AI to make or authorize battle plans or lethal decisions.

This is where the limits of the technology become existentially relevant.

AI systems do not know. They match. They generate outputs from learned patterns in data. This can work impressively well where the problem space is stable, the data is rich, and mistakes can be corrected. And very unstable where available information is adversarial, ambiguous, deceptive, and often unprecedented. In war, the available information is incomplete, the opponent actively tries to manipulate what is seen, and the consequences of being wrong are irreversible. This is exactly where AI systems are least accurate.

A drone or autonomous weapon with authority to strike without human authorization would not be "making a judgment" in the human sense. It would be applying learned patterns to a situation it does not understand. It would not know whether surrender is genuine, whether a civilian is being coerced, whether a signal is a decoy, whether escalation is strategically disastrous, or whether the correct decision is not to strike at all.

There is no reliable dataset that can teach a machine when it is legitimate, proportionate, strategically wise, and morally defensible to kill. Hypothetical scenarios are not experience. Simulations are not war. Pattern recognition is not command responsibility.

Prussian Generalfeldmarschall Helmuth von Moltke the Elder wrote that no plan of operations extends with certainty beyond the first encounter with the enemy’s main force. The point remains central: war destroys assumptions. Any system that depends on learned regularities is therefore at its weakest precisely when judgment matters most.

AI may help classify, recommend, prioritize, or predict. But that is not the same as exercising military judgment. And it is certainly not the same as bearing responsibility for life and death.

So whoever gives AI the power to make lethal decisions and removes human oversight has not understood the technology. They have mistaken pattern matching for judgment.

This is the real danger.