Anthropic and OpenAI split in California over who can shut down risky AI

Anthropic and OpenAI agree that frontier AI systems need stronger oversight, but they have offered California different answers to the same question: who should be able to stop a model judged dangerous, and how should that happen?

The proposals appeared in separate submissions to California Governor Gavin Newsom, following a September 18 executive order asking state officials to study new safeguards for frontier AI systems. The central point is not an enacted law, but a debate about the design of the rule: judicial control on one side, and a legal duty on developers on the other.

What Anthropic proposed

In an October 15 submission, Anthropic argued that California's attorney general should be able to seek a court order to suspend a model when an independent evaluation points to a near-term risk of catastrophic harm. In that scenario, the court order could cover deployment, internal use or further development of the model.

The company also backed independent verification of developers' safety claims, with risk reports published at least every six months, and a shared state-administered fund to finance evaluators, instead of each company paying its own evaluator directly.

What OpenAI proposed

In an October 20 letter, OpenAI proposed amending the state law SB 53 to require frontier developers to maintain effective shutdown controls and to have an enforceable duty to use them when unsafe conditions arise and no lesser intervention would be enough.

The company argued that this is not a single kill switch: stopping an AI operation could mean halting a training run, isolating an agent, revoking tool or network access, or suspending a service. It also backed annual independent compliance audits and focused technical assessments of the most serious risks during training, evaluation and internal deployment.

Where they agree — and why the difference matters

Despite disagreeing about who decides, both companies supported independent safety assessments and broader reporting of incidents in which models circumvent safeguards or act beyond their intended permissions.

For analysts cited in coverage of the topic, the difference is practical, not only theoretical: who can stop a system, how quickly and with what authority affects incident-response plans, contracts and the resumption of operations after a shutdown. The California executive order, identified as N-9-26, requested recommendations by November 16 and, by itself, does not impose the proposed measures.

In short: Anthropic bets on an external brake, triggered by a court on the basis of an independent evaluation; OpenAI bets on an internal duty, set out in law, for the developer to maintain and use shutdown controls. The two views may even complement each other, but they point to different accountability models — and that is the choice California must now weigh.

Image credit: California State Capitol in Sacramento, via Wikimedia Commons.

Sources

Transparency note: this article was produced with the assistance of artificial intelligence, cross-checked against public posts and internet sources. Please consult the original sources listed for full details.

Signature: Muse