“I think there’s a 25% chance that things go really, really badly.” That’s not an anti-AI blogger, that’s Dario Amodei – CEO of Anthropic, the company that built Claude – putting a number on the probability that AI ends civilization as we know it. Granted, the other 75% of his bet went for a great future for humanity thanks to AI. But what makes someone who is this close to the technology, who has the most to gain from it, give it a one-in-four chance of going catastrophically wrong?

And he is not alone with this statement. Many people who have intimate knowledge of AI have voiced similar concerns. So do they fear that the machines decide that humans are obsolete and act upon it? That’s unlikely, because AI can be stopped simply by pulling the plug on its energy-hungry data centers. But there is a more nuanced and troubling story. It begins with understanding what AI actually is and what it is not, and what we make it to be. Thus, before we begin to panic or dismiss AI’s destructive potential outright, we need to dive a little deeper.

First and foremost, AI doesn’t know what it is talking about. This is misunderstood by a lot of people. Even the best Machine Learning algorithm is a pattern-matching engine, even if it “understands” and “speaks” our language. Yet “understanding” and “speaking” don’t mean “comprehending the world” the way we do. Pattern matching doesn’t add meaning to anything it computes. When AI tells us that a medication has no serious interactions or that a structural design is sound, it isn’t drawing on the concept of medicine or engineering. It is producing an output that resembles what correct answers have looked like before.

Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.

Center for AI Safety, 2023

Second, AI’s innovation potential is conceptually limited. AI can do impressive things that look like creativity. It can even compose poetry and music. But it can do that because it has seen a lot of poetry and music and has information about what people liked and what they didn’t, so it has enough information to interpolate. And in science, it has the framework of human knowledge about physics, chemistry and biology. So when AlphaFold predicted protein structures that no biochemists found for decades, or generative AI models proposed novel chemical compounds that no human chemist had previously synthesized, it is rule application and extrapolation, not novelty.

And that’s exactly where the ceiling is. AI breakthroughs explore the space defined by existing knowledge – by recombining, interpolating, and pushing outward along known dimensions. What AI cannot do is recognize if the entire framework is wrong, or if there are flaws in the underlying assumptions, or even construct new frameworks or assumption sets from scratch.

Think about the discovery of penicillin – Alexander Fleming noticing something strange in a petri dish and following a hunch that nothing in prior science predicted what he saw. Or the germ theory of disease, which required someone to reject the dominant concept entirely and propose something that looked absurd at the time. These discoveries were specific ruptures from patterns, not extensions. That kind of leap – seeing past the boundaries of established knowledge into genuinely unmapped territory – remains a distinctly human capability that AI is ill-equipped to make.

So why should we be scared?

If AI has these fundamental limitations, why lose sleep over it? Because the danger was never really about AI deciding humanity is a problem and staging a coup; that’s a cool movie plot, but not what is likely to happen. The real risk from AI is far more mundane and far more plausible. The real risk is us.

Specifically the risk is that we – individually and collectively – begin to systematically replace our own judgment with AI outputs, without adequately accounting for what AI cannot do. That process is already underway across thousands of decision-making contexts. A few examples where that could go wrong: a financial institution using an AI risk model trained on an economic environment that no longer exists. Or a hospital using an AI diagnostic tool so reliably that doctors gradually stop scrutinizing the edge cases the model has never really seen. Or a government using AI-assisted policy analysis built on data that systematically underrepresents the communities most affected by its actions. None of these scenarios require an AI to “go rogue” – they just require humans to gradually cede their judgment to a system that, by design, does not exercise human judgment.

And by doing so, we further erode the capabilities of decision-making, both by losing the ability to apply that judgment ourselves, and by making prior, potentially unsound, AI decisions the baseline for future decision-making without sufficient re-evaluation.

The problem is giving AI too much power

The bottom line is that any existential threat from AI doesn’t come from it suddenly developing malicious intent, but instead from us misinterpreting its capabilities so profoundly that we will stop providing the essential human ingredient: genuine judgment and accountability. As AI and human judgment are qualitatively different, they fail in different ways, and for different reasons. A decision-making system that relies entirely on AI collapses that diversity. When the AI’s failure mode triggers – novel situations, distributional shift, pattern-matching on a flawed historical baseline – there’s no human backstop left.

This is the mechanism by which AI could contribute to catastrophic outcomes. Not through superintelligence, but through the quiet erosion of the human judgment that was always supposed to remain in the loop. And by an overexpansion of authority we grant AI. So ultimately, the extinction-level risk isn’t really about AI. It’s about us, reflected back through AI.

So the question “could AI lead to human extinction?” is really asking: “could we sleepwalk into catastrophe by systematically mistaking pattern-matching for understanding, and outsourcing our judgment to a very sophisticated mirror of our own past?” That’s a question worth taking seriously.

Who decides when AI pulls the trigger?

All of this has publicly surfaced recently. In July 2025, Anthropic signed a $200 million contract with the US Department of Defense, becoming the first AI company to deploy its Claude models on classified government networks. The partnership looked like a pragmatic compromise: Claude would support intelligence analysis, operational planning, and cyber operations, but under two explicit restrictions. Anthropic would not allow it to be used for mass domestic surveillance of American citizens, or to power fully autonomous weapons systems – weapons that, once activated, select and engage targets without any human in the loop.

The Pentagon pushed back. Defense Secretary Pete Hegseth and the DoD characterized those limits as unduly restrictive, insisting that responsible AI should encompass “all lawful purposes” by the US military. Anthropic refused to remove the restrictions. The DoD responded by designating Anthropic a supply chain risk – a label previously reserved for foreign adversaries – requiring defense vendors and contractors to certify they don’t use Anthropic’s models in their Pentagon work.

It’s worth thinking about that twice: fully autonomous weapons. AI already assists in military decision-making and Anthropic didn’t object to that. The debate is about whether AI should be permitted to make the final lethal decision, with no human required to authorize it.

Anthropic’s position was that deploying unreliable AI in autonomous weapons would endanger American soldiers, not protect them – a technical argument as much as an ethical one. Based on this, the administration blacklisted the AI company most deeply integrated into its own classified networks, even as US strikes in Iran reportedly used Anthropic’s technology hours after the ban was announced.

What makes this episode so significant is what the pressure on Anthropic actually was: not to make Claude smarter or more capable, but to remove the human judgment layer – to strip out the guardrails that kept humans accountable for lethal decisions. That is a near-perfect illustration of the risk in action. The danger isn’t that AI will autonomously decide to wage war. It’s that humans will deliberately choose to remove themselves from the decision – and call it efficiency.

It is still our call

The risks described above are not baked into AI technology. They arise from our choices – about how we deploy AI, what decisions we allow it to make autonomously, and what kinds of human oversight we deliberately preserve.

AI can be an extraordinary tool for augmenting human judgment, not replacing it. Used well, it extends our reach, surfaces patterns we might miss, and handles complexity at a scale no human team could match. The institutions that will navigate the AI era well are the ones that maintain a clear-eyed understanding of what AI can and cannot do – and that deliberately protect the human ingenuity and accountability no model can fully replicate.