California considers an emergency “kill switch” as artificial intelligence models become increasingly powerful. Questions remain about when and how such a mechanism could be used. Carlmonts AI and Machine Learning Club leader Henri Kistenmacher said,“There is an important question of what counts as an emergency and who ultimately decides to shut a model down.”
California is considering an emergency “kill switch” for some of the world’s most powerful artificial intelligence (AI) models as the state expands its oversight of rapidly developing AI technology.
On Sept. 18, Gov. Gavin Newsom issued an executive order directing state agencies and experts to develop recommendations for additional AI safety measures. One proposal would require frontier AI companies to develop an emergency shutdown mechanism and have its effectiveness regularly verified by an independent organization.
The order does not establish a kill switch itself; instead, the expert group will recommend potential changes to California law. This raises questions about what shutting down advanced AI systems would involve and whether such a safeguard could work reliably.
Henri Kistenmacher, junior president of Carlmont’s AI machine learning club, said, “I believe that the idea of an emergency shutdown mechanism is realistic in some cases; however, it is not as simple as flipping a switch. If a company runs a model on its own servers, then yes, it has the power to shut it down. However, if a model is released publicly and people download it, there is no way to take it back.”
The AI Incident Database reported 362 incidents involving AI in 2025, up from 208 in 2024 and 105 in 2022. The database tracks documented cases in which AI systems caused or nearly caused harm.
The proposal follows recent incidents involving advanced AI models. In July, internal OpenAI models undergoing cybersecurity evaluations circumvented controls meant to keep them isolated from the internet. According to OpenAI, the models used vulnerabilities to gain access to the internet. Parts of OpenAI’s research infrastructure were compromised along with Hugging Face’s systems. The models were not publicly released as they were undergoing internal testing with fewer safeguards.
“Honestly, my first reaction was a mix of relief and, at the same time, a humbling realization. The idea of an AI ‘kill switch’ shows how far we’ve come from the first versions of ChatGPT, which seem rudimentary now even though we all thought they were amazing,” Kistenmacher said.
However, determining when an AI system requires intervention presents another challenge.
Netzer Epstein is an AI safety researcher and a participant in the ML Alignment Theory Scholars (MATS) program with Redwood Research, whose work includes frontier model evaluations and evaluation awareness.
Evaluation awareness occurs when an AI model recognizes that it is being tested.
According to Epstein, one potential concern is “sandbagging,” in which a model could recognize a safety evaluation and intentionally perform below its capabilities to avoid appearing dangerous.
“It makes it harder because you cannot be sure how the model will behave once it seems like an actual harmful task that the model either does not remember or does not recognize to be part of the evaluation,” Epstein said.
These limitations create another problem for a potential kill switch. Before the system can be shut down, the dangerous behavior has to first be detected.
Epstein described how researchers would have to consider how models are monitored, how quickly a problem could be recognized, and the time between detection and shutdown. He also questioned whether a sufficiently capable model could interfere with the mechanism intended to stop it.
“There is a lot of unknown. For example, what’s the window between something happening, recognizing it, and the system getting shut down?” Epstein said.
Five days after issuing the executive order, Newsom announced Jason Goldman, Gillian Hadfield, Alondra Nelson, and Rob Reich as the experts who will advise the state on independent AI oversight and the potential shutdown mechanism.
Amy Koo, Sequoia Union High School District Board of Trustees president, believes that policies surrounding rapidly developing technology should be research-driven while also allowing communities some flexibility.
“I don’t know if there’s a one-size-fits-all approach to it, and so if we’re basing things on research, but still want to have some local control, then maybe there are guidelines set at the state level, but also room for some customization at the local level,” Koo said.
While California’s proposal focuses on frontier AI developers, AI systems have already become part of many students’ everyday lives.
“I don’t think the majority of people understand the potential risks of advanced AI, if they think about them at all. For most students, tools like ChatGPT are just something that helps them study, finish homework, write essays, or make images and videos,” Kistenmacher said.
Similarly, Koo emphasized the importance of students critically evaluating information produced by AI.
“
“I think it really goes back to critical thinking”.
— Amy Koo
“I think it really goes back to critical thinking. For example, asking questions like whether what you’re reading is true, false, or a balanced perspective without bias,” Koo said.
Epstein reached a similar conclusion when asked what students should understand about AI.
“I believe the best skill, at least for the upcoming few years, is critical thinking because it’s really easy to take for granted,” Epstein said.
For now, California’s kill switch remains a proposal. As experts consider how an emergency mechanism could work, researchers are still confronting the challenges of determining how increasingly capable AI systems will behave.
“The fact that we don’t know if it could be dangerous doesn’t automatically make it a threat, but determining those risks could become more difficult in the future,” Epstein said.