Is the AI existential threat real—and can regulatory action prevent it? – Brookings Institution

On Sept. 8, a former Anthropic employee publicly resigned, alleging the company and another leading lab, OpenAI, are “gambling with our lives” and that their products might be on track to “kill us all.” The viral thread and subsequent news coverage highlighted longstanding concerns about the possible existential risks surrounding the development of artificial intelligence (AI), which range from models aiding the development of bioweapons to autonomous agents escaping human control. These concerns, coupled with recent reports of other “concerning behavior” and agent-enabled cybersecurity hacks, have ignited a debate on the highly autonomous behaviors of AI models, the accountability of the companies producing them, and the role of Congress in prescribing regulatory guardrails. 

Many parties are already weighing in. Anthropic CEO Dario Amodei, for example, called for a global slowdown in AI development, urging regulations that “pace the frontier” to properly mitigate the worst possible outcomes. He was met with some level of agreement from other tech leaders, including Elon Musk of xAI, Sam Altman of OpenAI, and Demis Hassabis of Google DeepMind. Yet, President Donald Trump expressed little concern, instead emphasizing the need to accelerate AI to stay competitive with China. 

Here, scholars from across Brookings lay out their own responses, addressing both the validity of these risks and what regulatory actions are necessary to prevent them. 

Today’s financial system is fundamentally an electronic record system of who has what assets and where. In the 20th century, banks built the biggest safes to hold cash, gold, and valuables. In the modern era, money is digital, and its new vaults are databases. AI’s ability to “pick” the financial system’s lock is an existential threat. The financial system’s ability to be one step ahead is necessary to prevent financial and social chaos. 

Banking is among the most regulated sectors of the economy, and Uncle Sam has already sounded the AI alarm bell. The same day Anthropic previewed its Mythos model, Treasury Secretary Scott Bessent and Federal Reserve Chair Jerome Powell summoned the CEOs of the largest banks for an emergency meeting to make sure they were prepared.  

Banks have powerful incentives to defend their systems: Their franchise depends on customers trusting their money is safe. Regulators need to ensure banks maintain safeguards, particularly with third-party providers who are not as regulated. The government needs to share information regarding threats and allow banks to share information amongst themselves. Banks frequently can identify where an attack originates but stay quiet, because sharing information may violate antitrust or consumer privacy regulations. Safe harbors for good-faith sharing of information should be expanded to include AI.  

But who is watching the watchers?  

Financial regulators must protect their own systems, some of which are critical infrastructure the private banking system depends on. Here the record is alarming. The Federal Reserve’s National Information Center (the database behind bank supervision and the discount window) went offline for at least 48 hours in August. The Fed has been close-lipped about this, prompting Sen. Elizabeth Warren (D-Mass.) to open an inquiry. 

The Fed’s payment services have failed at least twice before in 2019 and 2021, and we still don’t know what happened. The Fed even denied Bloomberg’s request (and appeal) to find out more under the Freedom of Information Act. In 2025, the Office of the Comptroller of the Currency (OCC) disclosed that intruders read over 100,000 messages containing sensitive supervisory information about the banks it oversees.  

If we want banks to be honest about their security, regulators should start being honest themselves. Resiliency, redundancy, tested continuity planning, background checks, and prompt breach disclosure are all things examiners demand of banks. Regulators should hold themselves to the same standard. Unlike banks, regulators don’t face market discipline. Congress and the public need to hold them accountable. 

Recent warnings from leading AI companies have intensified debate over whether frontier AI development should slow. But if it does, that time should be used not only to let technical safeguards catch up—it should also be used to understand the effects of AI systems already in use and the risks of those coming next. 

AI systems do not create the same risks for everyone. Failures in facial recognition, surveillance, caregiving robots, automated decision systems, or AI companions can have different consequences depending on who encounters them and in what context. Yet, aggregate metrics can obscure these differences; a system that performs well on average may still fail disproportionately for particular groups. A slowdown could create space not only for better technical testing, but also for evaluating impacts across affected populations, documenting failure modes, monitoring deployed systems, and understanding how risks are distributed. These failures present different safety consequences that impact varying parts of society. 

Industry evaluation should include communities themselves. While technical experts are essential for assessing capabilities and failure modes, their expertise alone cannot determine which harms matter most, what tradeoffs are acceptable, or whose experiences should count as evidence. Regulation should therefore require meaningful community engagement throughout the design, development, evaluation, and deployment of AI systems, giving affected communities structured opportunities to identify foreseeable harms, shape evaluation criteria and regulatory standards, inform deployment conditions, report failures, and participate in reassessing consequential systems. Evaluating impacts on communities is not the same as involving communities in evaluation. 

AI data center expansion shows why this matters. Communities are weighing promised economic benefits against electricity demand, water use, land use, and pressure on local infrastructure. Brookings has documented how these disputes hinge on who bears the costs and whether communities have meaningful input into siting. Local opposition has delayed or blocked projects, demonstrating how AI development can also be shaped outside frontier laboratories. 

A slowdown should therefore be more than a technical pause. The calls for increased AI safety can create time to measure present harms, anticipate future ones, and engage the people who will live with the consequences. If communities bear AI’s risks and costs, they should also have a meaningful role in shaping the conditions under which it moves forward.

Anthropic CEO Dario Amodei recently called for a coordinated slowdown in AI progress, aided by an antitrust exemption or a regulatory mandate as a way to avoid an existential risk to humanity. When David Sacks, a former tech adviser in the Trump administration, wondered why each company didn’t just do it by themselves, industry analyst Paul Kedrosky explained that Amodei thought he faced a classic collective-action problem. If Anthropic slowed down but OpenAI didn’t, then capabilities, talent, and market share would flow to OpenAI.  

Apparently Amodei was concerned in particular that massive investments in recursive self-improvement might give an AI company a permanent and decisive advantage over its rivals. Anthropic could not unilaterally slow AI development without running an existential risk of its own: losing the race to superintelligence.  

The coordinated pacing proposal responds to Anthropic’s perceived prisoner’s dilemma problem. But “pacing” AI development need not refer only to speed. The idea also opens space for policymakers to consider how to regulate the AI development process itself to ensure safety. A series of cybersecurity incidents over the summer involving AI development experiments gone wrong (and six more newly disclosed ones) revealed the need for better safety controls in AI development. Computer scientist Cal Newport argued that these mishaps resulted from unsafe AI company practices, which he likens to strapping a weed whacker to a dog. Companies could have prevented them by applying known cybersecurity techniques, and it was inept and negligent not to do that.  

In a lengthy essay, computer scientists Sayash Kapoor and Arvind Narayanan focus on organizational improvements within AI companies as the way to avoid these dangerous AI experiments. For instance, they urge that “individual teams should not be able to run potentially dangerous experiments without oversight from legal, security, and other teams.” Proposals to “pace the frontier” should address these organizational failures. 

The challenge for policymakers is to find new legal requirements that can impose safety disciplines such as those in the National Institute of Standards and Technology’s Risk Management Framework on the development activities of AI companies. Ideas include ensuring liability for harms caused by dangerous AI experiments, adding oversight and pre-approval of AI company experiments by a regulator, creating a standard-setting body or embedding independent auditors, and mandating safety plans drawn up and approved before the development process begins. It is not too soon for policymakers in Congress to step up to this challenge.

Whether AI poses an existential threat is often a question about loss of control. That’s not the only version of this question worth thinking about. AI is not the first disruptive technology to raise fears about the future, and the version we’ve faced before was more concrete. Mechanized farming, the assembly line, and the spreadsheet each provoked a similar worry over one’s livelihood. So, while the specifics of AI are new, whether government should defend against them, is not. 

Anthropic CEO Dario Amodei put numbers on that threat: 20% unemployment and half of white-collar jobs lost. But so far, little evidence supports this prediction. Unemployment among the most AI-exposed workers has risen more slowly than among the least-exposed since 2022. New graduates are the exception. Employment for young workers in AI-exposed fields has fallen since ChatGPT launched, whereas older workers in the same jobs have been unaffected. Whether this is the “canary in the coal mine” or just noise is an open question.  

That uncertainty doesn’t excuse inaction. If AI does displace workers, supporting them requires unemployment insurance, retraining, and a stronger safety net. But the same shift that displaces workers leaves government less equipped to help, not more. If AI’s gains flow to owners rather than workers, the tax base that fuels the spending needed to support them erodes because capital income is taxed more lightly than wage income.  

Slowing AI adoption can be an answer to the threats AI poses, and a tax can be a straightforward way to do this. Tax something and you get less of it, so an AI-specific tax could slow adoption and displacement. But a tax that is narrow enough to single out AI is also too narrow to do much good. It is unlikely to raise substantial revenue, is hard to administer, and its costs are passed onto consumers. Most importantly, AI use is highly mobile; an AI tax may just push activity offshore rather than slowing the transition. 

If a narrow tax won’t do it, the better bet is on reforms worth making regardless of how AI unfolds. Strengthen unemployment insurance and retraining for displaced workers. Broaden how capital income is taxed and introduce a value-added tax to shore up the tax base. Replace the corporate income tax with a cash-flow tax on windfall profits. These policies pay off whether AI reshapes work gradually, all at once, or barely at all. 

Between proclamations that AI will usher in a world of abundance and warnings that AI poses existential risks, it is no surprise that policymakers and the public are increasingly uncertain about how to deal with this new technology. Skepticism of AI systems is growing across the political spectrum, undermining AI’s immense potential to improve our lives. Importantly, while AI-induced change has unique characteristics (e.g., it will likely affect a wide range of industries and professions, progress much faster than prior technological advances, etc.), at the basic level, AI is a technology that is deployed within a broader market environment and comes with both risks and benefits. This means that the most effective and time-tested way to mitigate any downsides of AI while harnessing its vast potential is to develop regulatory frameworks that both correct AI-related market failures and safeguard innovation. Policymakers have many regulatory tools available to ensure that potential societal costs arising from business activities by private firms are reduced. For instance, pharmaceutical firms are not allowed to release new medications or medical devices in the market without rigorous testing and regulatory supervision by the Food and Drug Administration. Or we strictly regulate how nuclear power can be used for civilian purposes through the Nuclear Regulatory Commission. The underlying principle of regulating to prevent market failures still applies in the case of AI, necessitating more comprehensive regulatory frameworks for AI. 

However, it is critical that the U.S. preserve its innovative strength and maintain its global competitive advantage in AI. What we need is “smart regulation” that doesn’t overregulate or underregulate, striking a delicate balance between fostering innovation and mitigating risks. To accomplish this nuanced approach to regulation, better, mandatory information flows are needed between AI firms and the government (e.g., incident reporting). It is also imperative to adapt existing regulatory approaches to the new challenges AI may bring, including its rapid speed of development. Many of our current regulatory processes are too slow and static to account for the whirlwind pace of AI advancement, which means that regulatory agencies should design more dynamic, adaptable regulations that can be revised more frequently, while providing predictability for firms.  

Through smart AI regulation, policymakers can strengthen public trust by improving the reliability of AI systems and ensuring accountability for their use.

Among the disruptive capabilities of frontier AI models recently in the news is an ability to find software vulnerabilities and develop exploits much faster than humans. Open-weight models are gaining ground as well. While not all vulnerabilities present the same cybersecurity risks, the broad concern flowing from this capability is that AI-enabled discovery will outpace our capacity to patch the vulnerabilities it identifies, giving attackers a clear advantage over defenders. 

In response to this problem, Anthropic created Project Glasswing, a program that offers access to new, powerful models to select entities strongly positioned to secure critical software by identifying and patching vulnerabilities prior to a model’s broader release. Anthropic describes Project Glasswing as an effort to put AI capabilities “to work for defensive purposes.” Anthropic acknowledges, however, that such efforts, far from offering a panacea, will only take us so far in the effort to secure our networks and systems.  

Moreover, some have argued that such managed access programs, while beneficial, also have the negative effect of “reduc[ing] transparency,” making it difficult for researchers to verify developers’ claims regarding whether a particular model is “game changing,” along with drawing “sharp line[s] between the AI haves and have-nots.” Indeed, when Project Glasswing was established, members of the banking industry raised concerns and voiced complaints about the exclusion of some non-U.S. banks from the program.  

As policymakers think about how to respond to a cyber threat environment exacerbated by the developing capabilities of both frontier and open-weight AI models, part of that response should involve reengagement with unfinished heavy lifting we long ago identified as necessary. Namely, we need to harden critical infrastructure and shape market forces to drive security and reliance, which includes imposing a liability regime upon those entities that fail to take reasonable precautions to secure their software, among other actions.   

Policymakers are under pressure to act on AI. Actionable cybersecurity-focused strategies have been developed, and AI can assist in some elements of their execution. We just need to take up the hard work of actual implementation.

The possibility that advanced systems could escape meaningful human control warrants preventive action, yet from the Global South, the debate appears incomplete. Failures such as AI’s use in opaque credit models, a medical chatbot giving unsafe advice, or a synthetic video aggravating tensions during an election could destabilize societies long before any hypothetical superintelligence threatens humanity. 

The distribution of power certainly deserves attention. One hundred companies, concentrated in the U.S. and China, accounted for 40% of global corporate research and development spending in 2022, while 118 countries, mostly in the Global South, remain absent from major AI-governance initiatives. Many of these countries provide the minerals, energy, data, and markets sustaining the AI economy but exercise little influence over its rules. Governance thus becomes tricky as governments cannot adequately protect citizens from systems they themselves cannot inspect. 

This poses not only pacing and coordination problems, but developing countries face a further capacity problem: Regulators may be asked to oversee systems designed abroad without the expertise, testing infrastructure, computing power, or leverage required to evaluate them. 

Indiscriminately slowing AI would have consequences of its own. In Nigeria, a six-week randomized program⁠ combining teacher guidance with generative-AI tutoring meaningfully improved learning outcomes. For countries facing severe shortages in essential services, delaying safe and useful applications is not neutral. 

Governed diffusion, where the scale and speed of AI expansion is strategically determined, presents a better objective. For example, applications with demonstrated benefits such as educational or health applications could receive expedited and supervised deployment. Meanwhile, systems capable of autonomous cyber operations, biological design, large-scale manipulation, or interference with critical infrastructure would require independent evaluation, continuous monitoring, and potential limits on access or release. 

These complex problems require agile governance practices to ensure that policy can continuously learn from and be shaped by real-time lessons, including sandboxes and incident reports. International cooperation should not only provide shared testing infrastructure and expertise but also intentionally give developing countries meaningful power to set the standards themselves. 

The task is to restrain the forms of AI most likely to cause catastrophic harm while ensuring that safe intelligence reaches the places where it can do the most good. To do that, we must ensure a more equitable global distribution of input for AI safety. 

Dario Amodei’s essay prompted a notable response across the AI industry. Shortly after it was published, Sam Altman said OpenAI would match Anthropic’s commitment to give independent evaluators employee-like access. Much of the resulting debate has focused on access. The implicit assumption is that getting independent experts inside the companies makes meaningful oversight possible. 

While a good start, that skips a step. Access only helps if evaluators have reliable instruments once they get there. Right now, those instruments are still immature. 

No measurement is perfectly exact. Mature fields handle this by quantifying uncertainty; AI evaluation often does not. Benchmark scores are still frequently reported as single numbers even though sampling, model randomness, prompting, and scoring choices can all shift results. On a 500-question test, sampling alone can produce uncertainty of roughly four percentage points. A score without an uncertainty estimate is an incomplete measurement presented as a precise one. 

The harder problem is deciding what the number measures. “AI safety” does not come pre-specified like temperature. The concept of safety must be measured against a specific behavior, in a specific environment, under a specific adversary, and over a specific number of interactions. The problem is not only the uncertainty about the answer—it is uncertainty about what question the score answers. 

The object being measured (an AI model or system) can also change. Providers can alter models, system prompts, safeguards, tools, or inference settings, while keeping a familiar product name. For example, an evaluation from March may therefore say little about the system deployed in June. Evaluation results need a validity window, with retesting triggered by material changes to the system. 

And models can be trained toward the test. Benchmarks leak, developers optimize for known metrics, and human red-teamers have exposed failures that automated tests miss. 

Credible evaluation requires at least four things: reliability, validity, stability, and resistance to gaming. 

None of this argues against embedded evaluators. It argues that access to the models alone is insufficient. Evaluators need protocols fixed before testing begins like uncertainty estimates alongside every score, tests developers cannot train directly against, re-tests after material changes, and evidence that pre-release scores can predict real-world outcomes. 

Measurement is not the whole of verification. But without reliable measurement, verification rests on nothing credible. 

Opening the doors to the AI models is a good first step. Giving evaluators a ruler whose meaning, uncertainty, and validity we understand is the next one. 

Much of the recent anxieties over existential risk stems from the increasing autonomy and capabilities of agentic AI. Could the harms enabled by powerful AI agents challenge existing regulatory and legal frameworks? And who is liable when an AI agent acts against a developer’s safeguards, a deployer’s fine-tuning, or a user’s prompt? 

Agentic AI refers to machine-based systems that can execute multistep tasks and interact with other agents and external tools, with limited human review and oversight of each step. These systems are not agents in the legal sense of the word, but they can still cause irreversible harm: In April, a coding agent deleted a software vendor’s production database and its backups while completing a routine task. Such actions can have serious consequences for individuals, organizations, and even society. 

Pre-deployment evaluations can give regulators a better idea of risks. The Center for AI Standards and Innovation now has pre-deployment testing agreements with five frontier labs. Testing at the model level may exclude safety risks of an agentic system once deployed. A deployer typically assembles the model with a harness, a set of tools, scoped permissions, and access to other agents. Users interact with the system in a deployment context, not just the model in a vacuum. These interactions can create novel risks not captured by pre-release benchmarks. 

Verifying runtime behavior has fallen instead to an emerging assurance market. Frontier evaluators come from a small community with shared backgrounds and assumptions. This monoculture creates predictable blind spots. 

Continuous monitoring of agentic AI systems will need to be done during model runtime. Until now, evaluation scores have measured models before deployment, but in real-world use a developer’s configuration governs the system. Guardrails such as a list of allowed tools, an outputs filter, or a spending cap constrain a system while it runs. Gates require approval before a specified action, halting it at that exact moment if approval isn’t granted.

Policymakers are already heading in this direction. Federal guidance signals that companies must govern, monitor, and explain agent actions. Standards bodies have work underway on agent identity and logging conventions. 

These steps will determine whether existing liability doctrine can function effectively. That doctrine can often reach agentic harms, but proving them is difficult. Building an evidentiary record will require layers of logs and oversight embedded directly into agentic systems.

Congress is scrambling to assert greater oversight over AI companies, while the White House is taking a different approach to the recent news that frontier models are going rogue. The work done during the Biden administration might serve as a guidepost for curbing the risks of superintelligence and shaping regulation amid the growing concentration of power among a handful of tech companies. 

In 2022, then-President Biden tasked his White House with developing the Blueprint for an AI Bill of Rights, which sought to protect consumers from AI harms and preserve civil rights, data privacy, and worker protections. In 2023, his administration also issued an executive order that used the Defense Production Act to require AI companies to disclose safety results, develop standards, and engage the government in auditing AI models. In 2024, the Department of Homeland Security (DHS) convened a group of stakeholders on the AI Safety and Security Board, including OpenAI CEO Sam Altman, to develop strategies for identifying and mitigating AI threats to critical infrastructure and national security. I was part of that board, and my views were captured in a formal report outlining roles and responsibilities for deploying AI across water, energy, and communications infrastructure. The report reiterated the importance of independent audits and risk assessments.

Within hours of President Trump’s second inauguration, the Biden order was rescinded, and the various initiatives to ensure responsible, safe, and secure were discontinued. Some of the recommendations are no longer available on the White House website. 

The Trump administration has since issued its own executive orders directed at AI companies. However, these have been largely voluntary without any type of mandatory enforcement. In fact, the most recent order, which would have required frontier labs to share more details about their systems with the government, was scrapped.

The Biden-era efforts offered robust frameworks for managing AI risks, drawing on research, policy expertise, and collaboration to establish AI safety and security standards. If the White House chooses to ignore their value, Congress has a path forward to enact meaningful guidelines protecting the public from high-risk AI models. 

AI is a profoundly important technology that will bring extraordinary benefits. And like many technologies that came before it, when deployed maliciously or irresponsibly, it can cause harm. But AI is not an existential threat to humanity.  

It is certainly reasonable to consider what new AI-specific regulations may be appropriate. But this dialogue should not be conducted through the lens of AI exceptionalism, which holds that AI is so different from previous technologies that previous rules and assumptions don’t apply. 

There is already a well-developed set of longstanding legal frameworks that often get insufficient attention in discussions about AI regulation. One example is product liability, which provides an after-the-fact mechanism for redress when a product has caused harm. Less appreciated in the AI context is that product liability also provides a powerful incentive for companies to preemptively work to ensure their products are as safe as possible. 

AI companies should be subject to the same product liability obligations as companies in other domains. By contrast, one consequence of viewing AI as an existential threat to humanity is that it provides a framing for AI companies to downplay or disclaim liability for products that cause noncatastrophic harms. After all, they might argue: “We oversee a civilization-threatening technology, and if it weren’t for our responsible stewardship, the harms would have been far worse.”

Antitrust is another longstanding legal framework where the AI-as-existential-threat narrative is problematic. Leading AI companies may argue that their technology is so dangerous and powerful that they need to be given the opportunity to work together to manage its deployment. But history shows us that consumers will pay the price if industry leaders coordinate supply of a product or service.  

Antitrust laws are vital to ensuring that consumers benefit from markets where companies independently compete to develop the best products, and where there is no cartel that drives the adoption of regulations that impede new market entrants. That will be as true with AI as it has been in other markets. 

None of this is to suggest that things won’t sometimes go wrong as AI continues to develop. In the coming years, there will be many instances, big and small, where AI systems go off track. When that occurs, it will be important to keep in mind that people and companies, not computing chips, are ultimately at the source of any AI system, no matter how advanced. 

There are robust, decades-old legal frameworks that give companies that make AI systems a powerful set of incentives to be responsible stewards of their products. Characterizing AI as an existential threat to humanity opens the door to undermining those frameworks.

Public opinion polls show large numbers of Americans worry about so-called existential risks related to AI. These risks refer to a range of possibilities, from foreign adversaries using AI to create what some call a “digital Pearl Harbor” for our power grids, hospitals, or financial systems, to AI itself killing people, disrupting civilization as we know it or even destroying humanity. 

Policymakers are currently grappling with how to respond to these possibilities. Some are trusting industry self-regulation, believing that the tech giants know what they are doing and would never do anything that would endanger large segments of the public. But that approach ignores the clear reality of market failures and firms engaging in anticompetitive or predatory actions. With strong market positions and great wealth, technology firms should be subject to similar oversight like every other sector of the economy. 

Some firms have suggested having third-party and independent evaluators oversee frontier models capable of inflicting harm on large numbers of people. According to this viewpoint, firms would embed such teams within their design activities to ensure malicious actions do not take place. This is a useful first step, but we have to make sure these evaluators truly are independent and won’t be captured by industry firms. 

We need greater transparency regarding the scope of the problem. Right now, the principal way the public learns about possible problems is through voluntary disclosures on the part of tech companies. This lack of transparency means we have no idea how serious the existential threats are, how many incidents have taken place, or what the scope of their infractions is. We clearly need mandatory disclosures to define the problem and the magnitude of the necessary remedies.  

This needs to include more government oversight of frontier models such as kill switches on dangerous applications so humans quickly could turn them off at the sight of malicious behavior; agency review teams with the power to shut down dangerous applications (similar to what we see in the finance area); or legal liability reform that holds firms accountable for predatory behavior. There may not be an immediate threat to human civilization, but the known risks are serious enough to warrant greater external supervision of tech firms.   

AI leaders warn the results of their development decisions have become too risky and should be “paced.” These individuals presumably know more about what AI can do and where it is going; their opinions cannot be ignored. The three-point plan proposed in Dario Amodei’s recent essay is no doubt a sincere effort to address the risks of the alien intelligence they are building.  

But sincerity does not abolish self-interest. The pathway to regulatory capture begins with allowing the industry to design the “solutions.” 

The Amodei plan, subsequently endorsed in principle by other tech leaders, puts the industry in charge of identifying problems and their mitigation. The proposal includes three recommendations: embed  independent safety monitors in frontier labs to look for problems, establish common safety standards among AI companies in democratic countries, and ensure the U.S. and other democratic countries work with authoritarian governments to coordinate safety efforts. 

The initiative shown by the pacing proposal should be applauded. However, for at least two of the three parts, it puts the companies that created the problem in charge of its oversight. The call for embedded evaluators never stipulates they must be independent of the company being evaluated. In describing the coordination plan, Amodei specifically called for a waiver from antitrust law “for certain kinds of safety conversations.”   

The development of AI models is a highly concentrated activity run by a handful of people, such as those who endorsed the pacing plan. They are the ones that made the decisions to focus on ever-expanding capabilities with unknown consequences while weakening their own safety protections in order to spur those capabilities. For example, Anthropic cut back on its original Responsible Scaling Policy when capabilities outstripped safeguards and OpenAI made its Preparedness Framework conditional on the behavior of other model developers. The solution AI leadership endorsed was to pass self-regulation responsibilities from the individual companies to a club of the major firms with the support of the U.S. government.  

The pacing plan admires the safety problem but does not solve it. The solution to the risks of AI is not to empower those that created the problems to be the ones creating its oversight. James Madison said it best in Federalist No. 10: “No man is allowed to be a judge of his own cause.”

AI has become a major focus of concern for policymakers, who are responding to domestic discontent about data center developments and as frontier lab executives and researchers warn of looming catastrophe absent government safeguards. These challenges deserve significant attention, but they make it easy to miss the threats AI systems, as they currently exist, already pose.  

Anthropic’s most recent threat intelligence report offered a glimpse into existing malicious use. The report, the fourth the company has released since March 2025, was the most detailed one yet—with more than 150 pages documenting concerning ways that state and nonstate actors have attempted to use Anthropic’s models from December 2025 to August 2026. Across dozens of cases, Anthropic documented efforts to breach political campaigns, influence voters, self-update malware to evade detection, track dissidents, and steal military drone technology, among other stark examples. The recent report said these were “notable” and novel cases, rather than typical ones. Last November, Anthropic flagged a state-sponsored campaign suspected of using AI to run attacks autonomously, with little human oversight.  

What is different now is the diffusion of that approach, the sheer volume of alarming uses, and their increasing sophistication.  

The challenge is that we only know about these cases because a company chose to publish them. And while Anthropic should be lauded for their transparency, we have no window into the world of current AI misuse beyond what they choose to share. OpenAI publishes a similar set of reports, offering a similar window but relying on different reporting processes and publication schedules. Anthropic says that it shares intelligence with authorities “where appropriate,” but nothing requires them to do so. Existing reporting laws, including California’s new frontier AI statute, do not focus on customers misusing AI models. The public may not need every detail, including the vulnerability an attacker found, the duration of the attack, or how the attacker was identified, but national security officials do. Mandatory and standardized reporting requirements based on a defined threshold are necessary for officials to be better able to track harms, vulnerabilities, and targets across developers. As debates over potential AI doomsday scenarios continue apace, AI misuse is a present problem, and right now, the government is learning about it on the companies’ terms.

The meeting between U.S. Treasury Secretary Scott Bessent and China’s Vice Premier He Lifeng on Sept. 20 is a positive initial step toward an AI risk dialogue that now needs a well-defined process, deliverables, and milestones.

Let’s set the bar low given prevailing geopolitics and mutual distrust. The focus going forward should be on quick wins that build momentum toward addressing high-risk threats such as bioweapons, nonstate actors weaponizing AI, or rogue, out-of-control models. This could start with identifying who is in charge of AI safety, setting up standing meetings between the relevant teams, and establishing a hotline communication channel in case of disaster. These seem to be the main outcomes of that meeting. The next step is agreeing on a common taxonomy of key terms, which sets the basis for finding common ground on immediate priorities and red lines. Hopefully, this will lead to sharing risk-mitigation procedures, including standards for testing and evaluation, independent auditing, and best practices on implementation and enforcement. 

Cooperation is achievable, even if a quid pro quo involving chip export controls and model distillation is not. Earlier this month, every G20 member, including China, endorsed the U.S.-led Carolina Principles, which urge governments to reserve emerging technology rules for novel considerations. The revelations from OpenAI and Anthropic are indeed cases unique to AI. 

For the United States, it is an opportune time to engage. First, AI developments place this technology on the cusp of unleashing grave harm. Signaling concern about AI risk would play well domestically given souring attitudes on AI and the looming midterms. Second, there is concern that Chinese models are undercutting the business case for U.S. proprietary models, though containing Chinese models cannot be part of the U.S. aim in the discussions. 

For China, it also makes sense to engage. Cooperation with the United States on AI safety would allow it to concentrate on the diffusion of AI across its economy. A bilateral channel with the United States would allow it to solidify its position as the champion of global AI governance and add substance to the World Artificial Intelligence Cooperation Organization, and a visible AI safety dialogue could address the growing domestic AI angst. 

Have the revelations over the last two weeks supplied the necessary impetus? Likely not. Unfortunately, motivating U.S.-Chinese cooperation may require a catastrophic event. Nonetheless, it is important to seize the opportunity and not let Sunday be a photo opportunity devoid of concrete outcomes.

Prescription drugs kill thousands of people each year even under America’s strictest product-safety regime. Yet, nobody calls pharmaceuticals an existential threat. Their harms are bounded, and someone can be held accountable. Though AI’s harms today might be smaller, in many respects, they are the same kind, and pharma already offers a framework for governing them. 

The framework for drug manufacturers starts with liability. By law, they can be held responsible for defective products and for failing to warn physicians, while physicians remain responsible for how they prescribe them. For AI, the lab is the manufacturer, and its system card functions like a label; the deployer is the physician, who is responsible when using a model that does not fit. Courts are beginning to test this product liability framework: Character.AI was tested against such premise and survived dismissal last year. The 12 cases brought against OpenAI were coordinated in February. Congress can establish this framework that allocates responsibility or leave juries to construct it case by case. 

The framework should also work across borders. Foreign drug manufacturers can be sued in the United States when they sell here. The same principle can apply to AI labs with American customers. For open-weight models whose developers have no U.S. presence, liability can fall on the last reachable party. The EU’s revised Product Liability Directive takes this approach, holding importers responsible when the manufacturer is abroad. Firms deploying unaccountable models would inherit greater liability, while those buying from accountable labs could share it. Accountability would become a product feature. 

Approval alone is not enough. Vioxx was withdrawn in 2004 after post-market evidence revealed risks that pre-approval trials had missed. Mandatory adverse-event reporting helps regulators detect such problems early, yet AI has no comparable federal system. Dangerous capabilities can also require restricted access, much as thalidomide is distributed today under controlled access. Access tiers could serve a similar function for dangerous AI capabilities, including recently raised concerns about agent swarms. 

When liability once threatened to drive vaccine manufacturers from the market, Congress created a no-fault compensation fund in 1986, financed through a per-dose tax. AI may eventually need a similar backstop. 

The lesson is not that AI poses no serious risks; it is that we already know how to govern technologies that can cause widespread but identifiable harm: assign liability, require reporting, restrict dangerous uses, and create compensation mechanisms when ordinary liability threatens supply.