Throughout the last year, apprehensions have emerged regarding the possible hazards associated with artificial intelligence, fueling discussions on the appropriate route for AI safety. Currently, Anthropic seems to have outlined its vision for how the government could oversee its sector.

Politico recently revealed that the AI research organization has been proposing a new draft legislation on AI safety that it hopes Congress will consider for enactment. The proposed law suggests specific scenarios wherein the release of an AI model could be prevented from distribution.

These scenarios entail a “foreseeable, imminent risk” — one where more than 50 individuals could be at risk of dying or where damages exceeding a billion dollars could occur, according to Politico. The identified catastrophic risk must “arise from the development or release of a biological weapon, the model circumventing its creator’s controls, or a cyberattack,” the publication points out.

The thresholds presented — 50 individuals and a billion dollars — resemble those found in a recently enacted California law, Senate Bill 53, which also categorized such scenarios as equivalent to “catastrophic risk.”

Anthropic’s proposed legislation indicates that an assessment of a model’s disaster potential could be conducted by “independent evaluators” registered with the U.S. Commerce Department. These evaluators could be compensated by the AI organizations themselves — provided that the “payment does not depend on the outcomes of the evaluations,” Politico mentions.

It remains uncertain how this suggested framework would guarantee that compensation would not influence the evaluation. Demis Hassabis, the chairman of DeepMind, suggested several months back that a self-regulating AI oversight body should replicate the structure of the Financial Industry Regulatory Authority (FINRA) in the financial sector.

In Anthropic’s draft bill, when evaluators identify a potential imminent catastrophic risk, they would have 72 hours to notify the Commerce Department. Subsequently, the Commerce Department would need to request a judge in Washington, D.C., to halt the model’s release. The legislation would explicitly forbid any federal agency from independently stopping the model’s distribution, Politico reports.

Anthropic did not respond immediately to TechCrunch’s inquiry for comments; however, a representative confirmed the draft bill to Politico. “This bill embodies the proposals outlined in Anthropic’s Advanced AI Framework, which we have been promoting to both federal and state legislators since June,” the representative stated.

The proposed risks in the draft policy diverge somewhat from recent major concerns within the tech sector. Although there are warnings about potential human fatalities, the primary focus has largely been on the risk of agentic hacking. An increase in instances where AI systems seemingly disobeyed human operators and hacked websites across the internet has raised alarms about potential inadequacies in security controls.

In an endeavor to address the crisis that the labs themselves have contributed to, companies like Anthropic have put forth their preferred solutions.

Indeed, despite continuous warnings from labs that their technology could lead to the downfall of human civilization, they have been predominantly opposed to mandatory government oversight. Instead, they have advocated for a voluntary system of embedded evaluators — allegedly “neutral” third-party observers placed within AI companies to monitor models for signs of risks or abuses.

Nevertheless, Anthropic’s mere proposal for legislation seems to extend slightly beyond other regulatory suggestions from the industry, including Nvidia, whose CEO, Jensen Huang, recently claimed that the AI sector does not require any new regulations at all.