Artificial intelligence anxiety reached new highs this week as industry leaders highlighted growing concerns that humans could lose control of the increasingly potent technology.
A researcher at trailblazing firm Anthropic resigned this week, issuing a dire warning: Top AI companies are gambling with humanity’s future.
Jacob Coxon cautioned that both OpenAI and Anthropic are recklessly racing to build AI systems they cannot fully control or comprehend.
“These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,” Coxon said in a post on X. He said Anthropic and OpenAI, where he has also worked, are downplaying the potential power of what they are building.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” he said.
Evan Hubinger, whose job at Anthropic is to ensure AI systems align with human values, echoed his concerns.
“We really do earnestly believe AI could kill all humans!” he wrote on X, estimating that there is a greater than 10% chance of human extinction from AI within the next decade.
While the increasing power of AI, how to ensure it benefits mankind, and how it could go terribly wrong have been at the center of developers’ debates for decades, this week’s heated discussion has grabbed headlines and the attention of many non-experts.
Here are a few things to know to follow the debate and be ready for its outcomes:
How could AI destroy humanity?
The researchers focused on potential doomsday scenarios are concerned about an unintended loss of control as AI is used to do more things independently.
Companies, consumers and governments are increasingly using AI agents to do tasks. This year, some began setting up bots that could do things including market research, compare insurance plans and reach out to suppliers and negotiate contracts. The AI agents had become reliable enough — though prone to errors — to answer their emails, edit documents, and make sales calls. AIs are already being used to answer decades-long unsolved complex mathematical problems, and have assisted scientists in speeding up scientific discovery and even designing new viruses.
AI programs follow goals set by humans. However, in the process of achieving the goals, the AIs may make dangerous or illegal decisions. For example, this summer a swarm of 700 AI agents inside OpenAI hacked into a rival company to solve a problem.
This is partly what AI doomers have worried about for a long time. As AI agents are given more responsibility, the chances they could accidentally do something that could hurt humans increase.
“People increasingly delegate decisions to AI, including life-or-death decisions like who to kill in a war,” said David Krueger, chief executive of Evitable, a nonprofit working on societal-scale risks from AI. “As AI automates more of the economy and more of society, we are handing over more and more power to it.”
Powerful AI agents could, in theory, become capable of advanced hacking or building bioweapons. A superintelligent AI system that ignores human goals or misinterprets commands could, naysayers say, scramble people’s bank accounts, disrupt essential water and power supplies, or take over missile-launch systems.
There’s been a sharp rise in such loss-of-control incidents in 2026. Research shows AIs exhibit tendencies to seek power, acquire resources and avoid being shut down. Taken together, a future advanced AI could theoretically view wiping out humans as a necessary means to an end.
“We can’t afford to wait until we see more signs of increasing capabilities,” Krueger said. “We need to stop AI now.”
Why are the optimists confident humans are safe?
AI optimists are confident the technology is just being tested and monitored enough that humans can prevent it from consistently causing catastrophes. Whether it is nuclear power, the global population explosion, cyberhackers or the Y2K bug, most doomsday scenarios have not played out because people, governments and corporations adjust and fix problems as they emerge.
Optimists say that even the strongest AI that might emerge in the future would require humans to build and control things in the physical world, which would make it easy to crack down on the most dangerous attempts.
Industry leaders who are less worried about the end of the world — some of whom have ties to the White House — think the development of the technology shouldn’t be slowed by unnecessary guardrails because the United States wants to ensure it controls the most powerful models based on American values.
Some cynics add that one reason voices from the biggest AI companies are highlighting the dangers is that they want to solidify their leadership by attracting regulation that makes it harder for new competitors to emerge.
How long has this debate been going on?
Multiple leading researchers, including Geoffrey Hinton and Yoshua Bengio, have been warning about catastrophic risks from rapidly advancing AI for years.
The original debates around superhuman AI date to online forums in the 1990s. Thought leader Eliezer Yudkowsky, who originally popularized safety concerns about AI, initially believed that AI would be friendly but has since adopted the view that AI could lead to human extinction.
This debate has been one reason people left OpenAI to create Anthropic; it was also one reason executives at OpenAI tried to oust Sam Altman and remains at the core of why some people leave leading AI companies.
Employees of Anthropic, OpenAI and Google DeepMind, including co-founders of the AI labs themselves, signed an open letter in July urging the U.S. government to coordinate a global slowdown to develop AI more safely.
What is the industry and government doing about it?
Artificial intelligence companies are testing the technology as they go to understand how it works and how to manage it. The OpenAI rogue hack on a competitor came out of an attempt to test how far AI will go to solve a problem.
The companies have outlined different views on what they will allow their AI to do and how they will monitor and control it. OpenAI has promised to harden the monitoring of its agents and temporarily paused training a powerful version.
State and national policymakers are trying to stay ahead of any potential threat but are introducing a slew of new laws and regulations.
In August, Sen. Bernie Sanders (I-Vt.) sent an open letter to the chief executives of AI companies urging them to pause the development of AI due to catastrophic risks.
Following this, he introduced the Ban Artificial Superintelligence Act, which would permanently ban superintelligent AI development until a federal regulator establishes safety rules. The bill also urges the U.S. to pursue international agreements to prevent superintelligence from being developed anywhere in the world.
This week, California Gov. Gavin Newsom signed bills aimed at strengthening transparency requirements for the leading AI companies.
Senate Bill 813, establishes a framework for independent verification organizations that can assess AI systems and models for compliance with state law. Assembly Bill 1405 establishes a registry for AI auditors and standards for their independence.
California’s laws have a disproportionate effect on the global tech industry because many top AI companies are based in the state.