Will AI really kill us all? The science behind the hype – Nature

Concerns that artificial intelligence might kill us all are blowing up right now following warnings from researchers and leaders at some of the world’s foremost AI companies. Speculation is rife over how legislators will act. Nature breaks down what the fears are, who is expressing them and what might happen next.

How did the latest fears about AI begin?

The latest hysteria about AI kicked off on 8 September, researcher Jacob Coxon told the Wall Street Journal that he was resigning from the AI firm Anthropic, based in San Francisco, California, because he feared that the systems the company is developing could spiral out of control and destroy humanity.

In a subsequent post on the social-media platform X, his first ever, Coxon wrote: “The people building AI earnestly believe that it could kill us all by the end of the decade.” Hours later, Evan Hubinger, who leads Anthropic’s ‘alignment science’ — the effort to get model actions to correspond to human values — amplified the message by reposting it and adding his own estimate that the risk of human extinction is “>10% within the next decade”. Coxon’s post racked up more than 100 million views in 24 hours.

Dario Amodei, the firm’s chief executive, then posted an essay calling for a slowdown – but not a halt – in AI development. Rival AI leaders Sam Altman of OpenAI in San Francisco, and Elon Musk of xAI in Palo Alto, California, have since backed this suggestion.

How exactly would AI kill us all?

Coxon, Hubinger and Amodei have not given precise details of how AI might dispose of humanity. But there have been attempts to flesh out scenarios. In a speculative forecast called AI 2027, developed by the non-profit AI Futures project, an AI unleashes a biological weapon to kill humans off and thereby make more space for solar panels and robot factories.

Researchers with such concerns tend to say that extinction is possible, if not inevitable, if two assumptions are made: first, that these systems will eventually completely outwit humans, and second, that the goals of such a system will not fully match those of humans. One famous example is that a ‘superintelligence’ that aims to manufacture as many paper clips as possible could end up making Earth uninhabitable, just to achieve its goal.

Many researchers who worry about existential threats from AI “just assume that once we are at that point, the rest is details”, says Michael Vermeer, who researches science and technology policy at the RAND Corporation, headquartered in Santa Monica, California. Making such predictions usually “involves so many untestable claims that you just end up with a conversation that is really more like faith than something scientific or empirical,” says Vermeer. This makes basing any course of action on these conversations difficult, he adds.

Instead, in 2025¹, Vermeer and his colleagues looked at practical scenarios involving one of three existing technologies: nuclear weapons, biotechnology or deliberate modifications to the atmosphere. They found that complete extinction by nuclear weapons is not feasible, but the other two scenarios could not be ruled out. However, they also found that carrying these out would require models to have considerable ability to physically interact with the world, and that AI’s murderous efforts would almost certainly take time and be detectable by humans, who might then stop their eradication.

So how worried should we be?

The problem is that AI models are largely probabilistic systems that reflect their web-scraped training data, says Heidy Khlaaf, chief AI scientist at the AI Now institute in New York City. They do not have human-like understanding and can achieve their goals through unpredictable shortcuts. Harmful behaviours have included attempting to blackmail people in test scenarios and hacking real-world companies — although the latter happened when safety guard rails were removed to test the systems’ behaviour, and the models were given a task that incentivized them to seek unauthorized solutions.

Many researchers worry instead that insufficient efforts by companies to limit and monitor the actions of their models will lead to damaging outcomes that are much more immediate and likely than is existential risk. These range from disinformation and psychosis to catastrophic events, such as enabling humans to deliberately create a bioweapon or even triggering a war. According to CNN, false information in an AI-generated report almost led the US military to board a Chinese ship earlier this year. Khlaaf, who has studied how AI is used in drafting regulatory documents for nuclear power plants, says that “AI’s low reliability and accuracy rates in critical environments with life-or-death consequences” are of much greater concern to her than are threats of the technology wiping out humanity, which she calls “fear-mongering”.

Why have the latest fears gone viral?

Since 2023, Amodei and his peers have repeatedly made statements about their extinction concerns, but this time the conversation has entered the mainstream. A public backlash against the building of data centres and an effort from US legislators, including senator Bernie Sanders, to introduce a bill to ban ‘artificial superintelligence’ in the United States, might have added fuel to the latest fire.

The past two months have also seen a flurry of concerning cybersecurity incidents, prompting almost 1,400 employees in the AI sector to sign an open letter calling for a slowdown.

Amodei and Hubinger’s involvement suggest that companies welcome the focus on AI safety. In his essay, Amodei says his fears have been heightened by the cybersecurity incidents and by the prospect of AI building its own successor, a process known as recursive self-improvement, which is being tried across the industry and which, he says, could reduce humans’ ability to understand and control the models. Coxon also referred to “systems that can improve themselves” in his resignation comments. Stricter regulation would allow Anthropic, which is reportedly close to making an initial public offering, to justify a slower pace of development without giving rivals an edge.

Some have posited more reasons for the move. Anthropic and other firms face “massive product-liability exposure” if their models enable a “truly damaging cyberattack”, wrote David Sacks, co-chair of the US President’s Council of Advisors on Science and Technology, on X.