California Proposes Emergency Power to Shut Down Dangerous AI Models
California Governor Gavin Newsom has released an 85-page report, "AI Safety in 2026 and Beyond", recommending mandatory safety standards for large frontier AI developers, continuous oversight by state-licensed independent verifiers, and emergency shutdown powers for models posing imminent catastrophic risk. The proposals are recommendations, not law, and are meant to shape legislation when lawmakers return on December 7, 2026.
California Proposes Emergency Power to Shut Down Dangerous AI Models
California wants the authority to order a frontier artificial intelligence model switched off in an emergency — and to place state-licensed auditors inside the laboratories that build them. Governor Gavin Newsom released an 85-page report on November 18, 2026, laying out the most far-reaching state-level AI safety regime proposed anywhere in the United States.
The report, titled AI Safety in 2026 and Beyond and dated November 16, was prepared by the California Government Operations Agency together with the California Office of Emergency Services and an outside expert group. It answers Executive Order N-9-26, signed on September 18, which asked for proposals in four areas: safety standards, independent verification, critical incident reporting and emergency shutdown requirements.
It is important to be clear about what the document is. These are recommendations, not law. The report says it hopes to inform the legislature's discussions as lawmakers return to Sacramento on December 7, and any of its proposals would have to be written into bills, passed and signed before they took effect. Newsom leaves office in January after two terms, so the fate of the package will rest largely with the next governor and the legislature.
State-set standards, checked by licensed outsiders
The first pillar of the plan shifts the job of setting the safety bar from the companies to the state. California would set baseline safety, security and transparency standards that every large frontier developer must follow, from training through internal use to deployment. Licensed independent verification organizations, known as IVOs, would turn those standards into technical tests and update them as evaluation methods improve.
Every large frontier developer would then face ongoing oversight by a licensed IVO, with full access to the models, people, systems, training environments and records the verifier judges necessary. Unredacted assessments would go to the state. The Government Operations Agency would license the IVOs and could revoke a license for insufficient expertise, independence or rigor.
If an assessor could not attest that a developer met the standards, the state could order corrective action after engaging with the company. Developers could challenge such a determination in court before it takes effect, apart from the expedited emergency process.
A surcharge on the verification fees developers pay would help finance publicly operated CalCompute capacity. Some of that computing capacity would be reserved for independent safety and security research and public oversight, linking an industry-funded surcharge to the infrastructure needed for external scrutiny.
The shutdown power — and how it would actually work
The headline proposal is emergency shutdown authority, and the report is careful about how it would work. Every large frontier developer would maintain a Risk Containment Protocol — a tested, documented ability to stop a model — certified as effective by a senior company official and checked continuously by an independent verifier.
The report is explicit that this is not a single on-off switch. The protocol must show the developer can intervene at the level of compute, agents, internal access and deployment, and coordinate with cloud providers and power suppliers. Testing would address ways shutdowns can fail, including a model copying its weights to obstruct containment or an outside actor gaining access to shutdown controls. For open-weight developers, the containment and shutdown requirements would apply only to deployments under their control, because they cannot shut down copies hosted by others.
If a developer refuses to act on credible reason to believe there is an imminent catastrophic risk or a major loss-of-control event, state officials could order an immediate shutdown. The order would require the narrowest workable intervention, take effect at once and remain in force during judicial review, although a developer could seek a temporary restraining order. It would be lifted when the danger is no longer imminent.
Faster incident reports and a watch on AI building AI
The report would also tighten how quickly the state hears about trouble. The deadline for most critical safety incident reports would fall from 15 days to 48 hours, while the 24-hour requirement for imminent catastrophic risks would stay. The reporting regime would cover attempted model-weight theft even without death or bodily injury, all internal or external loss-of-control incidents, and a new near-miss channel. Whistleblower protections would extend to former employees and contractors.
The rules would also reach further than training scale alone. Models capable of actions that could create catastrophic risk would be treated as frontier models for incident reporting even when trained with fewer than 10^26 integer or floating-point operations — a capability-based trigger meant to close a gap in the current law.
On the question of AI systems that automate their own development, the report stops short of an immediate limit. Companies automating AI research and development would report quarterly how their compute is divided among training, research experiments, inference for AI-research agents, safety and security work, external customers and unattributed activity, with IVOs verifying the categories. Recursive self-improvement — in its strongest form, an AI system independently developing its successor without human input — would be subject to standby cap authority. Officials could impose a time-limited, periodically reviewed cap on automated AI research when it exceeds a developer's committed thresholds or advances faster than its safeguards and evaluations, creating a material risk of catastrophic harm. The cap would be lifted after an IVO verified that adequate safeguards were in place.
Building on SB 53 — and the open questions
The package builds on laws California has already passed. Newsom signed SB 53 a year ago, the first US law governing frontier AI development, and SB 813 and AB 1405 this autumn created a framework for IVOs and a registry of AI auditors. What changes is the shift from voluntary to mandatory: the state sets the bar, and licensed outsiders check it.
The report notes that no federal law requires AI companies to follow safety standards, submit to independent testing or shut down a dangerous model, and calls voluntary commitments not enough. It leaves the thresholds, IVO licensing rules and CalCompute surcharge for a bill to set.
If passed, California would be the first US state with the power to order a frontier AI model switched off and to cap AI that improves itself. None of the large developers had publicly responded to the report by Wednesday evening. Whether the proposals become law now depends on the legislature that returns on December 7 — and on a new governor taking office in January.
Sources
- Unite.AI — Newsom Proposes Mandatory Frontier AI Oversight and Shutdown Powers, November 18, 2026
- California Government Operations Agency — AI Safety in 2026 and Beyond (report to the Governor, November 16, 2026)
- CASRAI — California Urges Mandatory Frontier AI Oversight, November 18, 2026
- AI Frontier Post — California proposes emergency shutdown power for frontier AI models, November 19, 2026
- MLQ.ai — Newsom report would let California order emergency shutdowns of frontier AI models, November 19, 2026
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0