Can AI lead us into the apocalypse?
Can AI end humanity?
Renowned AI experts attribute a significant chance to AI that it could lead to the end of humanity. Is that a realistic prediction or doomsday talk? Short answer: it's not AI that's the problem, it's us.
"I think there's a 25% chance that things go really, really badly." That's not an anti-AI blogger, that's Dario Amodei - CEO of Anthropic, the company that built Claude - putting a number on the probability that AI ends civilisation as we know it. Granted, the other 75% of his bet went for a great future for humanity thanks to AI. But what makes someone who is this close to the technology, who has the most to gain from it, give it a one-in-four chance of going catastrophically wrong?
And he is not alone with this statement. Many people who have intimate knowledge of AI have voiced similar concerns. So do they fear that the machines decide that humans are obsolete and act upon it? That's unlikely, because AI can be stopped simply by pulling the plug on its energy-hungry data centers. But there is a more nuanced and troubling story. It begins with understanding what AI actually is and what it is not, and what we make it to be. Thus, before we begin to panic or dismiss AI's destructive potential outright, we need to dive a little deeper.
First and foremost, AI doesn't know what it is talking about. This is misunderstood by a lot of people. Even the best Machine Learning algorithm is a pattern-matching engine, even if it "understands" and "speaks" our language. Yet "understanding" and "speaking" doesn't mean "comprehending the world" the way we do. Pattern matching doesn't add meaning to anything it computes. When AI tells us that a medication has no serious interactions or that a structural design is sound, it isn't drawing on the concept of medicine or engineering. It is producing an output that resembles what correct answers have looked like before.
About this post
In this article we review claims from key experts on how AI could create catastrophic outcomes for humanity. We evaluate this claim based on the functioning of AI and on our use of the tools.
Key takeaways
No, AI is not by itself going to end the world as we know it anytime soon. However, if we begin to use AI in a way that is not appropriate to its abilities, we soon might defer decisions to AI that should always be made by humans.
The text was written by a human and submitted to an AI system for final review, such as checking grammar, typos, or logical consistency - please click for more information
"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
Center for AI Safety, 2023
Second, AI's innovation potential is conceptually limited. AI can do impressive things that look like creativity, it even can compose poetry and music. But it can do that because it has seen a lot of poetry and music and has information about what people liked and what they didn't, so it has enough information to interpolate. And in science, it has the framework of human knowledge about physics, chemistry and biology. So when AlphaFold predicted protein structures that no biochemists found for decades, or generative AI models propose novel chemical compounds that no human chemist had previously synthesized, it is rule application and extrapolation, not novelty.
And that's exactly were the ceiling is. AI breakthroughs explore the space defined by existing knowledge - by recombining, interpolating, and pushing outward along known dimensions. What AI cannot do is recognize if the entire framework is wrong, or if there are flaws in the underlying assumptions, or even construct new frameworks or assumption sets from scratch.
Think about the discovery of penicillin - Alexander Fleming noticing something strange in a petri dish and following a hunch that nothing in prior science predicted what he saw. Or the germ theory of disease, which required someone to reject the dominant concept entirely and propose something that looked absurd at the time. These discoveries were specific ruptures from patterns, no extensions. That kind of leap - seeing past the boundaries of established knowledge into genuinely unmapped territory - remains a distinctly human capability that AI is ill-equipped to make.
So why should we be scared?
The pattern matching limit
There are objections on our assertion that "mere pattern matching" is a true limitation for AI and it all relies on the definition of intelligence. If we define intelligence narrowly as representation of knowledge or problem-solving ability in a cleanly known and documented domain, AI outperforms any human expert.
But that this is the essence of intelligence is not true. Human intelligence comprises the contextualization of content into a world image formed over years, which - correct or not - guides decision. AI doesn't have that construct internally and will not have it as long as it's built based on vectors mapping the past.
If AI has these fundamental limitations, why lose sleep over it? Because the danger was never really about AI deciding humanity is a problem and staging a coup; that's a cool movie plot, but not what is likely to happen. The real risk from AI is far more mundane and far more plausible. The real risk is us.
Specifically the risk is that we - individually and collectively - begin to systematically replace our own judgment with AI outputs, without adequately accounting for what AI cannot do. That process is already quietly underway across thousands of decision-making contexts. A few examples where that could go wrong: a financial institution using an AI risk model trained on an economic environment that no longer exists. Or a hospital using an AI diagnostic tool so reliably that doctors gradually stop scrutinizing the edge cases the model has never really seen. Or a government using AI-assisted policy analysis built on data that systematically underrepresents the communities most affected by its actions. None of these scenarios requires an AI to "go rogue" - they just require humans to gradually cede their judgment to a system that, by design, does not exercise human judgment.
And by doing so, we further erode the capabilities of decision-making, both by losing the ability to apply that judgment ourselves, and by making prior, potentially unsound, AI decisions the baseline for future decision-making without sufficient re-evaluation.
The problem is giving AI too much power
The bottom line is that any existential threat from AI doesn't come from it suddenly developing malicious intent, but instead from us misinterpreting its capabilities so profoundly that we will stop providing the essential human ingredient: genuine judgment and accountability. As AI and human judgment are qualitatively different, they fail in different ways, and for different reasons. A decision-making system that relies entirely on AI collapses that diversity. When the AI's failure mode triggers - novel situations, distributional shift, pattern-matching on a flawed historical baseline - there's no human backstop left.
This is the mechanism by which AI could contribute to catastrophic outcomes. Not through superintelligence, but through the quiet erosion of the human judgment that was always supposed to remain in the loop. And by an overexpansion of rights we grant AI. So ultimately, the extinction-level risk isn't really about AI. It's about us, reflected back through AI.
So the question "could AI lead to human extinction?" is really asking: "could we sleepwalk into catastrophe by systematically mistaking pattern-matching for understanding, and outsourcing our judgment to a very sophisticated mirror of our own past?" That's a question worth taking seriously.
Who decides when AI pulls the trigger?
All of this has publicly surfaced recently. In July 2025, Anthropic signed a $200 million contract with the US Department of Defense, becoming the first AI company to deploy its Claude models on classified government networks. The partnership looked like a pragmatic compromise: Claude would support intelligence analysis, operational planning, and cyber operations, but under two explicit restrictions. Anthropic would not allow it to be used for mass domestic surveillance of American citizens, or to power fully autonomous weapons systems - weapons that, once activated, select and engage targets without any human in the loop.
The Pentagon pushed back. Defense Secretary Pete Hegseth and the DoD characterized those limits as unduly restrictive, insisting that responsible AI should encompass "all lawful purposes" by the US military. Anthropic refused to remove the restrictions. The DoD responded by designating Anthropic a supply chain risk - a label previously reserved for foreign adversaries - requiring defense vendors and contractors to certify they don't use Anthropic's models in their Pentagon work.
It's worth thinking about that twice: fully autonomous weapons. AI already assists in military decision-making and Anthropic didn't object to that. The debate is about whether AI should be permitted to make the final lethal decision, with no human required to authorize it.
Anthropic's position was that deploying unreliable AI in autonomous weapons would endanger American soldiers, not protect them - a technical argument as much as an ethical one. Based on this, the administration blacklisted the AI company most deeply integrated into its own classified networks, even as US strikes in Iran reportedly used Anthropic's technology hours after the ban was announced.
What makes this episode so significant is what the pressure on Anthropic actually was: not to make Claude smarter or more capable, but to remove the human judgment layer - to strip out the guardrails that kept humans accountable for lethal decisions. That is a near-perfect illustration of the risk in action. The danger isn't that AI will autonomously decide to wage war. It's that humans will deliberately choose to remove themselves from the decision - and call it efficiency.
It is still our call
The risks described above are not baked into AI technology. They arise from our choices - about how we deploy AI, what decisions we allow it to make autonomously, and what kinds of human oversight we deliberately preserve.
AI can be an extraordinary tool for augmenting human judgment, not replacing it. Used well, it extends our reach, surfaces patterns we might miss, and handles complexity at a scale no human team could match. The institutions that will navigate the AI era well are the ones that maintain a clear-eyed understanding of what AI can and cannot do - and that deliberately protect the human ingenuity and accountability no model can fully replicate.
Why making war decisions with AI is a particularly bad idea
The idea of using AI in war deserves particular scrutiny, particularly when it comes to allowing AI to make or authorize battle plans or lethal decisions.
This is where the limits of the technology become existentially relevant.
AI systems do not know. They match. They generate outputs from learned patterns in data. This can work impressively well where the problem space is stable, the data is rich, and mistakes can be corrected. And very unstable where available information is adversarial, ambiguous, deceptive, and often unprecedented. In war, the available information is incomplete, the opponent actively tries to manipulate what is seen, and the consequences of being wrong are irreversible. This is exactly where AI systems are least accurate.
A drone or autonomous weapon with authority to strike without human authorization would not be "making a judgment" in the human sense. It would be applying learned patterns to a situation it does not understand. It would not know whether surrender is genuine, whether a civilian is being coerced, whether a signal is a decoy, whether escalation is strategically disastrous, or whether the correct decision is not to strike at all.
There is no reliable dataset that can teach a machine when it is legitimate, proportionate, strategically wise, and morally defensible to kill. Hypothetical scenarios are not experience. Simulations are not war. Pattern recognition is not command responsibility.
Prussian Generalfeldmarschall Helmuth von Moltke the Elder wrote that no plan of operations extends with certainty beyond the first encounter with the enemy’s main force. The point remains central: war destroys assumptions. Any system that depends on learned regularities is therefore at its weakest precisely when judgment matters most.
AI may help classify, recommend, prioritize, or predict. But that is not the same as exercising military judgment. And it is certainly not the same as bearing responsibility for life and death.
So whoever gives AI the power to make lethal decisions and removes human oversight has not understood the technology. They have mistaken pattern matching for judgment.
This is the real danger.