Anthropic urges court orders to block AI models posing serious risks

Anthropic wants government officials to be able to ask a court to stop the release of an AI model when a review finds that it poses a serious risk to the public. The proposal appears in an October 15 letter to California Governor Gavin Newsom, according to a Reuters report published on October 28.

The Claude developer confirmed the letter's authenticity through a spokesperson. Newsom's office had not immediately responded to Reuters' request for comment. The recommendations are a policy proposal, not a new California rule or an announcement that any particular model has been blocked.

A court would decide whether to block a release

Anthropic's approach would give officials a route to seek judicial intervention before a dangerous model reaches users. It places a court between the government's request and a restriction on release, rather than treating a developer's own safety assessment as the final decision.

The letter followed Newsom's decision in September to convene a panel of AI experts to develop safety recommendations. OpenAI also sent recommendations to the governor on October 20, Reuters reported. Politico first reported details of Anthropic's submission.

Why Anthropic favors targeted controls over blanket kill switches

The company drew a distinction between preventing a risky release and trying to shut down an entire AI model after losing control of its actions. It warned that requirements for universal model-level kill switches could give officials a false sense of reassurance.

Instead, Anthropic supported narrower interventions, such as disabling a problematic AI agent or ending a particular customer's access to a model. That distinction matters because an agent carrying out a harmful task, an abusive customer's account and the underlying model are different points at which a developer might intervene.

These are Anthropic's arguments about the design of safeguards. The letter does not establish that targeted controls will always work, or that government officials have adopted the company's preferred approach.

Several oversight proposals remain unsettled

Reuters reported that a bipartisan group of U.S. senators had discussed legislation in September that would allow the government to block certain unsafe AI models, with companies able to challenge a decision in federal court. That proposal had not advanced after Congress left for its pre-election break.

A separate AI Kill Switch Act proposed in the House in July would give officials authority to order the shutdown of models that threaten human life or the economy. It is also a legislative proposal, not an enacted power described by the report.

Anthropic's letter adds a specific enforcement mechanism to a broader debate: who should judge a model's risks, when intervention should happen, and how narrowly restrictions can be applied without leaving serious dangers unaddressed.

Source

Reuters: Anthropic tells California governor courts should be able to block risky AI models