Catastrophic Benchmarks: Anthropic Pitches Federal AI Safety Bill Requiring 50 Casualties for Government Action

Artificial intelligence research lab Anthropic drafted a federal safety bill and pitched the legislation to lawmakers in Washington, outlining high thresholds for government intervention against risky AI models. The draft proposal establishes specific catastrophic risk scenarios that allow federal authorities to block or halt a model’s public distribution. Under the proposed rules, the government can only step in if an AI system poses a predictable and imminent threat of killing or severely injuring more than 50 people, or causing over $1 billion in property damage.

The proposed legislation defines catastrophic risk around three specific scenarios: the creation or release of biological weapons, AI-driven cyberattacks, or an autonomous system breaking free from developer controls. These metrics mirror specific thresholds established in California’s recently passed Senate Bill 1047, which similarly defines catastrophic risk based on massive financial loss or mass casualties. Under Anthropic’s draft, any harmful incident or safety failure falling below these exact limits would lack a clear legal mechanism for federal suspension.

Anthropic’s bill relies on independent evaluators registered with the U.S. Department of Commerce to analyze frontier models. AI companies could pay these independent evaluators directly, provided those payments remain independent of the final safety score. If an evaluator identifies an imminent catastrophic threat, the bill gives them 72 hours to notify the Commerce Department. The Commerce Department would then need to petition a federal court judge to issue an emergency injunction, as the draft explicitly denies federal agencies discretionary power to halt distribution on their own.

Routing all regulatory enforcement strictly through the Commerce Department represents a significant structural design choice. By centralizing oversight inside a single federal agency and requiring court approval for any model suspension, the draft limits independent oversight from other regulatory bodies. This setup prevents local or state agencies from enforcing separate reporting requirements or blocking releases independently.

Critics and industry observers raise concerns over relying on third-party auditors paid by the very tech companies they inspect. Earlier proposals from tech leaders suggested creating self-governing regulatory bodies modeled after the financial industry’s FINRA framework. However, skeptics argue that relying on company-funded auditors fails to create meaningful independence, leaving big tech labs to shape their own enforcement rules.

The theoretical threats outlined in Anthropic’s draft focus on extreme, world-ending catastrophes like biological weapons or rogue superintelligence. Yet everyday safety concerns across the tech industry look quite different. Current real-world risks focus primarily on rogue agentic behavior, website hacking, automated fraud, and software systems defying human instructions on the open internet. Setting high thresholds for federal action leaves everyday digital harms and automated security breaches completely unaddressed by federal safety rules.

An Anthropic spokesperson confirmed that the draft aligns directly with the company’s Responsible Scaling Policy, which leadership has advocated to federal and state lawmakers since June. While some tech leaders, including Nvidia CEO Jensen Huang, argue against passing new AI laws altogether, Anthropic’s proposal offers a structured middle ground. However, establishing high damage thresholds ensures that corporate AI labs retain significant freedom to launch advanced models without constant federal interference.