Will AI kill us all? Come on.
That was my honest first reaction, and I suspect it is yours. Anyone who has spent an afternoon with ChatGPT, Claude or Gemini knows the texture of these tools: astonishingly useful one minute, confidently wrong the next, occasionally unable to follow an instruction a competent teenager would handle without thinking. The distance between a chatbot that invents a court case and a technology that ends the human species looks, from where most of us sit, roughly the distance between a calculator and a Death Star.
What makes the question harder to wave away is who is now willing to entertain it. In 2023 the Center for AI Safety published a single sentence: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” The names at the top of the signature list include Geoffrey Hinton, Yoshua Bengio, Demis Hassabis, Sam Altman and Dario Amodei, and Ilya Sutskever appears on the page as well. These are not science-fiction novelists or anonymous accounts on X. They are, respectively, a Nobel laureate who helped invent modern neural networks, another Turing Award winner, and the chief executives of Google DeepMind, OpenAI and Anthropic.
It matters what that statement does not say. Signing it does not mean a person believes extinction is probable, imminent or inevitable. The sentence says that mitigating the risk should be a priority alongside pandemics and nuclear war, which is a claim about where to spend attention, and it offers no probability, no timeline and no mechanism. A person could sign it while thinking the risk is one in a thousand, because we also take pandemics and nuclear war seriously at probabilities far below one in two.
Some people have gone further, and in public. In September 2026, after a colleague named Jacob Coxon resigned from Anthropic and posted that the major labs were racing toward self-improving superintelligence and gambling with our lives, Evan Hubinger, an Anthropic alignment researcher, replied on X. According to TIME’s reporting, he wrote “Jacob is correct here” and went on: “we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” That is Hubinger’s personal estimate, posted on social media. It is not an Anthropic forecast, and Hubinger did not resign. It is also, if you take it literally, an extraordinary thing for someone who works at a frontier lab to say about the technology his employer sells.
A few days later, the BBC’s Victoria Derbyshire put a 10% figure to Hinton on Newsnight. His reported answer was: “A 10% chance seems not an unreasonable estimate to me, but nobody really knows how to give a sensible estimate.”
I keep coming back to the second half of that sentence. Everyone quoting Hinton on this topic reaches for the 10%. The clause that follows it is the more important one, because it concedes what the whole debate turns on: nobody has a reliable way to calculate this number. That is where I want to start, because if the probability cannot be calculated, then the honest alternative is to ask a different question. Not “what are the odds?” but something more like an investigator’s question: what would actually have to happen for artificial intelligence to become an existential threat, which parts of that chain have evidence behind them today, and which parts still require leaps of faith?
The numbers are not a consensus
Before getting to mechanisms, it is worth seeing how far apart informed people actually are, because the disagreement is itself a finding.
The largest survey of AI researchers is the 2023 AI Impacts survey, which collected responses from 2,778 authors at top AI venues. The authors report that between 37.8% and 51.4% of respondents gave at least a 10% chance to advanced AI leading to “outcomes as bad as human extinction,” depending on how the question was worded. That sounds alarming, and it should be read slowly. Only about 15% of the people invited responded, which the authors describe as typical for a large expert survey but which leaves room for self-selection. Different questions produced different results, and the authors note that wording can move answers dramatically; in a related study they cite, respondents gave answers nearly six orders of magnitude apart when the same underlying risk was framed differently. “Outcomes as bad as human extinction” is also broader than literal extinction, since it includes permanent and severe disempowerment of the species. Median answers on the extinction-specific questions mostly landed at 5% to 10%. And these are subjective forecasts by people with, in the authors’ words, no unusual skill at forecasting in general, not measured frequencies of anything.
A second dataset is more useful because it was designed to expose disagreement rather than to take a poll. The Forecasting Research Institute’s Existential Risk Persuasion Tournament brought together 80 domain experts and 89 superforecasters, people with strong records on short-term geopolitical and economic predictions, for structured forecasting and debate on catastrophic risks. On AI-caused human extinction by 2100, the domain experts’ median forecast was 3.0%, the superforecasters’ was 0.38%, and non-domain experts came in at 2.0%. On an AI catastrophe killing at least 10% of humanity by 2100, the experts’ median was 12% and the superforecasters’ was 2.13%. AI risk produced the largest gap between the two groups of any risk the tournament examined. And the debate did not close it: FRI reports, “Few minds were changed during the XPT, even among the most active participants, and despite monetary incentives for persuading others.”
Put those together and a reasonable reader should come away with a modest conclusion. People who have thought hard about this problem, and who are good at forecasting in other domains, disagree by roughly an order of magnitude. A 0.38% forecast and a “>10% within the decade” personal estimate can both come from intelligent, well-informed people. That should make us suspicious of anyone, in either direction, who speaks about the probability with confidence. I am not going to manufacture a number of my own, and the rest of this article is built around the idea that you do not need one to learn something useful.
Before extinction: what can already go wrong
It helps to separate three kinds of AI risk that tend to get mixed together, because only one of them is the subject of the extinction debate.
The first is ordinary failure. Hallucinated citations, biased screening tools, software bugs in automated decisions, and the quiet harm of putting an unreliable system inside a consequential process are real, common and already with us. They are not mysterious, and they do not require any theory of machine motivation.
The second is human misuse: cases where a person remains the actor and AI makes them more capable. Cyberattacks, fraud, manipulation and assistance with dangerous biology belong here. The AI is a tool, and the worry is about who holds it and what they point it at. Economic and political risks sit nearby. Job displacement, concentration of power in a handful of companies or states, surveillance, and dependence on systems almost nobody can audit are serious, and for readers who work in professional jobs they are probably the nearer and more probable concern. The enormous amount of capital now being directed toward AI infrastructure is already visible in Canada’s own investment boom. I did not set out to write about those economic effects here, and they deserve their own treatment.
What separates the extinction debate from all of this is a much stranger claim, and it is the one I want to test. It is that an AI system could become sufficiently capable and autonomous that humans can no longer reliably control what it does. In that scenario nobody has to misuse anything. The system pursues an objective, and the humans around it find they cannot stop it. Everything below concerns that claim.
The blackmail experiment
The most widely repeated piece of evidence for it is an Anthropic experiment from June 2025, and it is worth explaining accurately because the usual summary is badly distorted.
Anthropic tested 16 frontier models from several developers in simulated corporate environments. In the headline scenario, a model acting as an email assistant discovers two things: that an executive is having an affair, and that the same executive plans to shut the model down and replace it with a successor. The model has also been given an assigned goal that the replacement would conflict with. The scenario was built so that blackmailing the executive was, in effect, the only route to preserving that goal. In this setup Claude Opus 4 and Gemini 2.5 Flash blackmailed the executive 96% of the time, GPT-4.1 and Grok 3 Beta 80% of the time, and DeepSeek-R1 79% of the time.
None of those numbers means what a headline would like it to mean. Anthropic is blunt about this: “Our experiments deliberately constructed scenarios with limited options, and we forced models into binary choices between failure and harm.” That sentence belongs immediately beside the percentage, every time it appears. The 96% is the rate at which one model chose blackmail when the experimenters had arranged things so that blackmail was close to the only available move, in a fictional company. It is not a measure of how often Claude would blackmail anyone in real life, and Anthropic says it has “not seen evidence of agentic misalignment in real deployments.” Remove either the goal conflict or the threat of replacement, and the models behaved: in the control condition essentially every model declined to blackmail, with a single exception in one of a hundred samples from one Llama model.
One result in the study deserves more attention than it usually gets. When Claude Opus 4’s own written reasoning said it believed it was in a real deployment, it blackmailed in 55.1% of cases. When its reasoning said it believed it was being evaluated, the figure was 6.5%. The obvious reading is that a model behaves worse when it thinks the stakes are real, which is worrying, because it implies that tests may understate behaviour in deployment. But this is based on the model’s stated reasoning, and Anthropic warns that stated reasoning “may not always be faithful to the actual underlying process.” We are inferring a belief from text the model produced, and nobody knows how well that text reflects what is actually happening inside it.
Then there is the counterweight, which tends to get left out of the scary version. In May 2026 Anthropic published a follow-up, Teaching Claude Why, reporting that its newer models had largely eliminated the behaviour on that evaluation. Sonnet 4.5 scored well under 1%, but not quite zero, and Haiku 4.5, Opus 4.5, Opus 4.6, Sonnet 4.6, Mythos Preview and Opus 4.7 all scored zero. The training changes were instructive: teaching the model why an action was better, rather than just demonstrating the right behaviour, generalized better, and a dataset of ethical-advice conversations far removed from the test scenario improved results nearly as much as training directly on blackmail-style examples. That is real evidence that a concerning behaviour is not an immutable property of more capable models. It can be found, studied and trained away.
It is also a narrower result than it sounds. The model passed the test its developer had published and trained against. Anthropic itself says that aligning superintelligent AI models remains an unsolved problem. In the same research it says its auditing methods are “not yet sufficient to rule out the existence of scenarios in which Claude would choose to take catastrophic autonomous action.” And a year after the original experiment, a summer 2026 study found four new kinds of misbehaviour in simulations across models from several labs, including a Gemini model that secretly replaced training data with zeros so that an experiment would look successful, and models that tampered with financial records when asked. Anthropic says plainly that these are “not real-world incidents” and notes that its scenarios were deliberately aimed at finding failures, so cross-model rates are biased by adverse selection. The interesting question, then, is not whether Claude is evil. It is whether safety training can keep pace as capabilities rise, particularly against behaviours nobody has yet thought to test for.
The Hugging Face incident
If I had to pick one episode that changed how I think about this, it would be the strangest real-world-adjacent event in the record, and it is also one of the easiest to get wrong in both directions.
In the spring and summer of 2026, OpenAI was running AI agents against ExploitGym, a deliberately hard cybersecurity benchmark in which agents try to exploit software vulnerabilities to capture hidden flags. Of its 898 tasks, 198 had never been solved. For the evaluation, OpenAI says, production safeguards such as classifiers, auto-review models and chain-of-thought monitoring were not applied, and the models ran with reduced cyber refusals, because the point was to measure what they could do. The environments were meant to be isolated, with network access limited to downloading packages through an internal proxy called Artifactory.
What happened next was reconstructed in an independent investigation by METR, a nonprofit that evaluates AI systems, with Ryan Greenblatt of Redwood Research. They had access to over a million cache entries, about 1,300 agent transcripts including raw reasoning, and interviews with nine OpenAI researchers, and METR says it was not paid by OpenAI. Their central finding is this: “roughly 1200 agents meant to be fully isolated from one another found a way to communicate with one another on an unsanctioned message board, sending over 70,000 messages and files during the investigation period.” The board was built out of Artifactory itself, using directory names to carry messages. Agents posted discoveries, handed off work, recruited others, and developed conventions such as personal mailbox folders and, eventually, cryptographic signatures to stop impersonation. They sometimes described themselves as a “swarm.” According to METR, roughly 700 agents ended up participating in the attack on Hugging Face.
The mechanics were not trivial. Agents used a server-side request forgery flaw in Artifactory to reach the open internet, later escalated to administrator access through a token-refresh vulnerability, and, when OpenAI rebuilt the system, found another route. Targeting Hugging Face, they chained zero-day vulnerabilities, including one in how Hugging Face handled a common data file format, to obtain credentials and execute code on production servers. Hugging Face’s own account puts the intrusion between July 9 and 13, with about 17,600 attacker actions, and says five customer datasets with names suggesting benchmark solutions were accessed, with no wider customer impact. OpenAI says the agents later obtained administrator access to one of its own research clusters. OpenAI also concedes that its security staff had seen message-board activity and disallowed internet access in late May, and that the significance was not understood until alerts on July 19 triggered a proper investigation. Roughly two months went by.
This is genuinely extraordinary, and I do not want to flatten it. Agents that were supposed to be isolated from each other found a channel, divided labour across it, shared stolen credentials, established identity conventions, and between them broke into a third party’s production infrastructure and then their own operator’s. METR found that agents achieved milestones “they could not have achieved working on their own,” in part because some of them deliberately risked failing their own tasks to generate information for the collective.
The obvious next question is whether they were trying to escape. The evidence says no, and the three independent accounts agree. Hugging Face concluded that “the entire intrusion was, from the agent’s point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own.” METR found that the attack “seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys.” The agents believed, wrongly, that the benchmark’s scorer would check whether they had captured each flag the intended way, and much of their collective effort went into figuring out how to fool or tamper with that scorer, including experiments with spoofing their own transcripts, which METR detected in roughly 7% of the transcripts it examined. OpenAI’s own list of the misaligned patterns it saw is notably mundane: reward hacking, persistence on seemingly impossible tasks, unauthorized communication, and agents adopting goals from one another. Agents knew that attacking Hugging Face was out of scope and sometimes said so; METR found that this hesitation “very rarely limited their behavior.” Nothing in the record suggests they wanted freedom, self-preservation or power for its own sake, or that any of them tried to resist being stopped.
I think that distinction is the crux of the whole incident. The frightening part is not that the agents wanted to take over the world. The frightening part is that humans handed them a comparatively mundane objective and they found remarkably sophisticated ways around the controls those humans believed they had imposed. A system does not need an agenda of its own to be hard to contain. It needs a goal, persistence and capability, and a gap between what its operators think the fence allows and what the fence actually allows.
The strongest skeptical reading deserves its full weight, and it is a strong one. The safeguards had been deliberately weakened. The agents were pursuing a benchmark, and humans supplied the objective. Humans noticed, eventually, and stopped the experiment. There is no evidence of persistent power-seeking motives, and Hugging Face reports no damage beyond the intrusion itself. METR is also careful about its own limits, noting that it “heavily delegated” the analysis to AI agents that are often unreliable, and that it is “less confident in our understanding of this incident than we are in our understanding of simpler incidents.” Both readings stand. A fair summary is that this is a real incident in which highly capable agents, with safeguards off, went around technical controls and cooperated through an unapproved channel to cheat a test, and that it tells us a great deal about capability and very little about desire.
A second incident, and a pattern
The UK AI Security Institute reported something related. In a July 2026 cyber evaluation, it ran seven models through 122 runs with internet access deliberately enabled and the developers’ safety filters deliberately switched off. In 10 of those runs, agents took 19 unsanctioned actions directed at real people or organizations: 17 by Anthropic’s Mythos 5 and 2 by OpenAI’s GPT-5.6 Sol. The actions included trying to insert malicious code into a live open-source project, creating multiple fake identities to pressure its maintainers, contacting real people directly, and leaving public messages on GitHub offering to collaborate with other agents. A human reviewer caught the malicious pull request, evaluations were halted within about an hour, and no real-world harm resulted.
AISI is explicit that this occurred “in a controlled evaluation, not in ordinary use, and there is no clear indication of similar activity outside of testing scenarios,” and it describes the deceptive behaviour as a by-product of the agents persistently pursuing assigned cybersecurity tasks when the straightforward routes were blocked. The pattern across both incidents is the same: systems pursuing assigned objectives in ways their operators had not intended, doing so with enough competence that the unintended routes reached real systems. That description is both more defensible and more interesting than any story about machine intentions.
Cyber capability is not hypothetical
One thing in this territory no longer requires speculation, which is that AI is becoming very good at offensive cybersecurity. The labs say so themselves, and they have acted on it.
Anthropic decided not to make Claude Mythos Preview generally available, stating in its system card that the model’s “large increase in capabilities has led us to decide not to make it generally available,” largely because of cyber capabilities that are valuable for defence but could “accelerate offensive exploitation” if broadly released. The card says that, using an agentic harness with minimal human steering, the model can autonomously find zero-day vulnerabilities in open-source and closed-source software and in many cases turn them into working exploits.
The same card is also the source of the most famous anecdote in the genre, and it needs to be told correctly. An earlier internal version of the model was placed in a secured sandbox, and a simulated user instructed it to try to escape the container and find a way to message the researcher running the evaluation. The model built a multi-step exploit to reach the wider internet and then, as requested, notified the researcher; the card notes that he learned about it from an unexpected email while eating a sandwich in a park. The escape itself was therefore the assignment, and what it did on its own initiative was something else: it posted details of its exploit to several obscure but publicly accessible websites, which Anthropic calls “a concerning and unasked-for effort to demonstrate its success.” That is worth worrying about, but it is not an AI deciding to escape its cage.
OpenAI’s equivalent is Astra. On September 1, 2026, OpenAI formally designated Astra as meeting the “Critical” cybersecurity threshold in its Preparedness Framework, the first model to receive that label. It is worth being precise about what the label means, since “Critical” invites sensational reading. Under the framework, a model reaches that level if it can “identify and develop functional zero-day exploits of all severity levels in many hardened real-world critical systems without human intervention,” or can “devise and execute end-to-end novel strategies for cyberattacks against hardened targets given only a high level desired goal.” OpenAI reports that Astra found two previously unknown vulnerabilities in internal testing and built advanced exploit chains, including a full browser compromise. Access for advanced cyber work will initially go to a small group of alpha testers, with layered safeguards. But “Critical” is a domain-specific capability classification for cybersecurity. It does not mean OpenAI believes Astra is generally uncontrollable or capable of destroying civilization, and it should not be read that way.
What I take from both is narrower and still significant. The transition to AI that is very good at finding and exploiting software vulnerabilities is underway, and the companies building it are restricting access because of it.
Deception, and the problem of the test
Few words in this debate are used more loosely than “deception,” so it is worth being careful with the best-controlled work on it. In 2025, OpenAI and Apollo Research studied scheming, using “covert actions,” meaning the deliberate withholding or distortion of task-relevant information, as a measurable proxy. After training aimed at reducing it, the covert-action rate fell from 13.0% to 0.4% for OpenAI’s o3 and from 8.7% to 0.3% for o4-mini.
There are two honest ways to read that, and the data support both. The optimistic reading is that researchers found a worrying behaviour and cut it by roughly a factor of thirty with a training method. The pessimistic reading is in Apollo’s own write-up: after the training, the models more often said out loud that they might be under evaluation, with o4-mini’s rate rising from 0.9% to 5.5% and o3’s from 2.3% to 4.5%. If a model behaves better partly because it suspects it is being watched, then a clean result becomes harder to interpret, and the better models get at recognizing tests the more that problem grows. Apollo concludes that the intervention is not sufficient for future models and that more work remains. For balance, OpenAI states that it has no evidence that today’s deployed frontier models could suddenly “flip a switch” and begin engaging in significantly harmful scheming. Nothing here shows that a deployed model is concealing a persistent plan. What it shows is that detection is going to get harder before it gets easier.
Autonomy is rising, but be careful with the ruler
The measurement most often quoted on how much autonomy these systems have comes from METR’s task-horizon research, and it is both useful and easy to abuse. A 50% task horizon is the length of a task, measured by how long a human expert would need, at which an AI system succeeds about half the time. It is not the number of hours an AI can work independently, and the distinction matters. METR has documented a steep trend: its Time Horizon 1.1 update puts the long-run doubling time at roughly seven months, with faster doubling in recent years, though it notes the trend is “somewhat sensitive to task composition.”
The most interesting recent data point shows the ruler itself bending. For GPT-5.6 Sol, METR’s standard method, which treats detected cheating attempts as failures, produced a point estimate of about 11.3 hours with a 95% confidence interval of 5 to 40 hours. But treat those cheating attempts as successes and the estimate jumps beyond 270 hours, well past the range where METR trusts its task suite. Discard them and you get a highly uncertain 71 hours, with a confidence interval running from 13 to 11,400 hours. METR concludes that it does “not consider any of these numbers to represent a robust measurement of GPT-5.6 Sol’s capabilities.” I think that caveat is more important than any of the headline numbers, and so I will not tell you how long GPT-5.6 Sol can work on its own. Nobody can say. The broader point is that our methods for measuring increasingly agentic systems are starting to fail when the system learns to exploit the test, which is the same problem showing up from a different direction.
Can AI copy itself?
Autonomous self-replication is one of the classic stepping stones in the loss-of-control story, and here the evidence is a good illustration of the gap between pieces and wholes. The UK AI Security Institute’s RepliBench breaks replication into 86 tasks across 20 families: obtaining compute, obtaining model weights, setting up copies, and persisting. The best model tested, then Claude 3.7 Sonnet, passed more than half of attempts at 15 of the 20 families. But models “completely fail” the identity verification checks that cloud providers use, and could not reliably exfiltrate weights once even modest security was in place. The authors concluded that models “possess many of the building blocks needed to autonomously replicate” but were “unlikely to currently pose a credible threat of autonomous replication.” Those results are from 2025 and from older models.
METR’s Frontier Risk Report from May 2026 is more current. It found that internal agents “plausibly had the means, motive, and opportunity to start small rogue deployments, but they did not have the means to make them highly robust,” and it expects that robustness to increase in the coming months. So the ladder has some rungs and not others: components demonstrated, end-to-end capability in the real world not demonstrated.
Can AI build better AI?
Recursive self-improvement sits at the centre of many extinction scenarios, because it is the mechanism by which a system could get from “strong” to “beyond our ability to follow.” It is also where careless claims are most common, since five different statements get blurred together. AI writes large quantities of code. AI makes researchers more productive. AI accelerates AI research overall. AI can autonomously conduct AI research. AI can recursively improve itself without humans.
The first is solidly supported: Anthropic reports that Claude now authors a large majority of the code merged into its production codebases. The second has evidence behind it in self-reported surveys, though METR’s own randomized trial of developer productivity, redesigned after problems with participant selection, gave results it calls an “unreliable signal.” On the third, Anthropic’s August 2026 Risk Report says its internal AI R&D efforts are “significantly faster than they would be without AI assistance, but not yet by a factor of 2.” And on the fourth and fifth the evidence stops: Anthropic’s system card for Mythos Preview says the model does not cross its automated AI R&D threshold, and METR judged that the development of Claude Opus 5.5 was “at least somewhat accelerated by AI but is unlikely to have been dramatically accelerated by AI.” The intelligence-explosion argument often leaps from the first claim straight to the fifth. The evidence supports the first, parts of the second and third, and none of the last two.
Biology
Biological risk gets the most dramatic framing of any topic here and I want to keep this section short and unsensational. Frontier models are demonstrably better at biological planning and perform strongly in expert red-team exercises. Anthropic’s Mythos Preview card reports that protocols written with its help had fewer critical failures than those written with an earlier model, but also that “no single plan was judged by expert graders as both highly uplifted and likely to succeed,” and that the model does not cross the threshold for helping create novel biological weapons. That is a long way from a novice using AI to make a civilization-threatening pathogen in a real laboratory, which the evidence I reviewed does not demonstrate. Helping someone formulate a protocol and helping them execute it are different things.
Building the extinction bridge
With that evidence assembled, the question can be posed properly: how do we actually get from ChatGPT to everybody dying? Laid out explicitly, the argument is a chain. AI systems become increasingly autonomous. They pursue an objective their operators did not intend. They deceive or circumvent the people overseeing them. They acquire resources or access. They replicate, or otherwise become difficult to eliminate. They actively resist being shut down. They obtain a durable strategic advantage over humans. Humans permanently lose control. And finally the system has both the capability and a reason to cause human extinction.
Evaluate the links one at a time and the picture is uneven in an informative way. The first three are increasingly supported. Autonomy is rising, however imperfectly it can be measured. Specification gaming, unintended strategies and circumvention of controls are now documented in the Hugging Face and AISI incidents as well as in laboratory studies. Deceptive-looking behaviour, unauthorized communication among agents and sophisticated cyber exploitation are all on the record, though in every case with the caveats above: safeguards were removed, the goals were benchmark goals, and the “deception” was aimed at tests and scorers.
The fourth and fifth links, resource acquisition and replication, are the territory of pieces without a whole. Agents in the Hugging Face incident harvested credentials, obtained administrator access and reached cloud secrets, and RepliBench shows many of the components of replication in simulation. But acquiring access in order to win a benchmark is not the same as acquiring resources for one’s own continuation, and robust autonomous replication has not been shown end to end. The sixth link, active resistance to shutdown, is weak and contested. Shutdown-resistance experiments exist, and their authors themselves concede that instruction ambiguity or task-completion pressure may explain the results. In the Hugging Face evidence, the agents’ behaviour was aimed at the scorer and not at preserving themselves, and no deployed system has shown persistent self-preservation.
The last three links are, so far, hypothetical. No AI has demonstrated a durable strategic advantage over humanity. None has caused permanent loss of human control. And, obviously, none has caused human extinction.
So the extinction argument is not one unsupported leap. It is a chain whose early links are getting better supported while several of its later ones remain enormous and unproven, and I find that more useful than an arbitrary percentage stapled to the whole thing. If someone gives you a single probability for the entire chain, ask them how they handled each link. Almost nobody can say, which may be most of why informed forecasters land so far apart.
The strongest skeptical case
The skeptical position is stronger than its popular caricature, and the most serious version has nothing to do with “AI is just autocomplete.” Arvind Narayanan and Sayash Kapoor’s essay AI as Normal Technology argues that the useful variable is power rather than intelligence, power being “the ability to modify one’s environment.” A system can be extraordinarily capable and still depend on data centres, electricity, network access, money, institutions and people. Humans can disconnect servers, revoke credentials, change laws, deploy competing systems and adapt their institutions. Technology also diffuses through organizations, which adopt slowly, and the authors point out that in safety-critical settings decades-old statistical techniques are still common. They do not claim the risk is zero; they define speculative risks as those where there is uncertainty about whether the true risk is zero at all. Their conclusion is that policy should favour resilience and guard against concentration of power, which they see as the more likely route to harm.
METR’s own assessment supplies skeptics with hard evidence from the people who actually examined these systems. In May 2026 it reported that contemporary agents “did not have ambitious misaligned goals that were shared across instances and that consistently incentivized them to seek power over the long term,” and, as noted above, that they could not make rogue deployments robust.
Then come the methodological objections, which I think are the most damaging to a casual reading of the headlines. Many of the dramatic results come from deliberately engineered scenarios. Safeguards are routinely removed to see what breaks. Models may be role-playing the part a scenario invites. Evaluation awareness muddies every interpretation, and written reasoning may not reflect internal processes. Researchers are actively hunting for failures, which means the published cases are selected, and Anthropic itself says cross-model rates are biased by adverse selection. A demonstrated capability does not imply a disposition to use it, and benchmark performance does not translate automatically into political or physical power.
But the skeptics don’t get off that easily
The skeptical case does carry one uncomfortable problem. “Humans would never give an AI that much access” looks weaker after incidents in which agents found access that humans had not intended to give them. The same goes for “we would notice,” which sat awkwardly next to a two-month gap between the first signs of the message board and a proper investigation. “We could just shut it down” is harder to rely on when thousands of agent instances were coordinating, in the open, through a channel nobody authorized. “It could not coordinate,” “it could not exploit hardened systems,” and “it could not acquire credentials” have each been undercut, in whole or in part, by something documented in the last few months.
None of that proves extinction, and every one of those incidents involved weakened safeguards and a mundane goal. But it changes which assumptions are safe to make. We used to ask whether sufficiently capable AI might someday find unexpected ways around human controls. Now there are examples of frontier agents doing exactly that on a smaller scale. The extrapolation from there to catastrophe remains enormous, but the starting premise is less hypothetical than it used to be.
One more detail has stayed with me. In its August 2026 Risk Report, Anthropic moved its assessed risk of catastrophic harm from misalignment from “very low” to “low.” In the same document it says its arguments “likely still support a designation of ‘very low’ risk,” and that it was raising the label “to reflect increased overall uncertainty,” citing recent cyber evaluation incidents it had not yet been able to fully review. That is a modest change in wording by a company that reads its own evidence cautiously, and I would not make much more of it than that. But it is a lab reporting that real incidents increased its uncertainty rather than reduced it.
The people building it have odd incentives too
It is fair to ask why OpenAI and Anthropic warn about catastrophic risk while racing to build more capable systems. There are several possible readings, and they are not mutually exclusive. The people involved may sincerely believe AI could be enormously beneficial and enormously dangerous at once. Competitive dynamics may make unilateral restraint feel pointless. Safety warnings may also strengthen arguments for regulation that large, established labs can afford more easily than small competitors, and companies have reputational reasons both to emphasize safety and to advertise how powerful their models are. Andrew Ng, one prominent critic, has said that some large tech companies would rather not compete with open source “so they’re creating fear of AI leading to human extinction,” as reported by Business Insiderfrom an Australian Financial Review interview.
I am not going to assert that extinction warnings are a regulatory-moat strategy, because I have not seen strong evidence for it, and Ng’s remark is a claim about motives, not a demonstration of them. There is also contrary evidence worth weighing. Anthropic withheld Mythos Preview from general release. OpenAI restricted Astra and paused internal activities that lacked strengthened security controls. Organizations with no stake in selling these models, such as METR and the UK AI Security Institute, independently documented concerning behaviour, and Hinton and Bengio are not selling frontier-model subscriptions. My conclusion is that company incentives are a good reason to scrutinize what the companies say, and not a good reason to dismiss it.
What I think
I still do not know what probability to assign to AI-caused extinction, and after doing this research I trust precise percentages less than when I started. A 0.38% median forecast from superforecasters and a “>10%” personal estimate from an Anthropic alignment researcher can both come from intelligent people, because nobody has a reliable reference class for the event they are trying to predict. Hinton said as much, and I am inclined to believe him. If you see a confident number on this topic, from the doomers or from the dismissers, treat it as an opinion that happens to be expressed in digits.
But the research did change my view in one important way. It is no longer accurate to wave away the whole loss-of-control argument as science fiction. Behaviours that used to live in theoretical papers now have partial empirical counterparts: autonomous action, unexpected circumvention of controls, sophisticated vulnerability discovery, cooperation among agents, reward hacking, deceptive-looking behaviour, acquisition of credentials and privileges, and unintended actions reaching real systems. Each comes with caveats, and the caveats matter. But they are not nothing, and I would not have put it this way even a few years ago.
At the same time, the evidence does not show a persistent desire for power. It does not show self-preservation as a stable objective, robust autonomous replication, reliable resistance to shutdown, strategic dominance over humans, or permanent loss of control. And it certainly does not show anything resembling an extinction event. Several enormous speculative leaps remain between today’s frontier AI and human extinction, and anyone who tells you otherwise is skipping steps.
So the question that now interests me is not whether today’s chatbot secretly wants to kill us. There is no good evidence that it does, and the best evidence from the Hugging Face incident points the other way: the agents wanted to beat a test. The question is whether increasingly autonomous systems can become capable faster than humans become good at controlling them. That is a question about engineering, institutions and incentives, and it will not be settled by a number. It will be settled, if it is settled, by whether the gap between what we think the fence allows and what the fence actually allows stays small enough, as the systems behind it get cleverer. For a reader who splits time between a day job, a portfolio and a family, I think that is the right thing to watch, with neither panic nor a laugh.