Recommended Amendments to California’s AI Incident Reporting Provisions

Executive Order N-9-26 directs the Government Operations Agency, in consultation with the Office of Emergency Services (“Cal OES”), to recommend amendments to existing law. Among other things, the order calls for an updated definition of the “critical safety incidents” that frontier developers must report, covering loss-of-control incidents like the OpenAI–Hugging Face breach. We recommend amending the reporting provisions of the Transparency in Frontier Artificial Intelligence Act (“SB 53”), as set out below. 

SB 53’s current incident reporting regime has three major defects:

• The thresholds are too high or too uncertain: Three of SB 53’s four categories of “critical safety incident” require physical harm to have already occurred: theft of frontier model weights and loss of control are reportable only if they cause death or bodily injury, and the catastrophic-risk category only covers harm involving 50 deaths or serious injuries or $1 billion in damage. The fourth, deception by a model to subvert its developer’s controls, applies only where the behavior “demonstrates materially increased catastrophic risk,” a judgment a developer cannot make with confidence. Under the existing definition, incidents such as an agent escaping its sandbox and committing a cyberattack, or a foreign state stealing model weights, will often not be covered. 
• Reports provide insufficient information: For incidents that do qualify, a developer is only required to report the date, the reason the incident qualifies, whether it involved internal use, and a “short and plain” description of the incident. Amended reports are optional, and nothing in SB 53 authorizes Cal OES to require a technical investigation or a follow-up report. The quarterly summaries of internal-use catastrophic-risk assessments that large frontier developers send Cal OES (§ 22757.12(d)) capture some of what the critical safety incident definition misses, but only require submission of whatever assessments the developer chooses to conduct. A developer can therefore limit what Cal OES learns by assessing internal-use risk less thoroughly. 
• Information sharing is limited: Cal OES and the Attorney General may share incident reports with the Governor, the Legislature, the federal government, and “appropriate” state agencies (§ 22757.13(e)), but not with other developers exposed to the same failure and only arguably with other states. The public receives only an annual aggregate (§ 22757.13(g)). Sharing of § 22757.12(d) internal-use summaries is not authorized.

We recommend the following amendments to SB 53, in order of priority:

Expand the definition of “critical safety incident”: Amend § 22757.11(d) so that unauthorized access to, modification of, or exfiltration of unreleased model weights, a model’s circumvention of its developer’s controls, an autonomous cyberattack by a model, and damage to or disruption of critical infrastructure are each reportable when they occur, whether or not harm results. Critical safety incidents should also include any death, serious injury, or substantial property damage to which a model materially contributes within a catastrophic-risk category (CBRN, autonomous cyberattacks and other crimes, and loss of control) or through automated AI research and development. This would capture incidents like Hugging Face as well as comparably concerning loss-of-control “warning shots” in domains other than cyber. Crucially, the categories we identify are observable events. Objective categories are preferable because relying on a developer’s subjective judgment regarding difficult questions like whether an incident demonstrates materially increased catastrophic risk creates legal uncertainty. The expanded definition should be accompanied by a duty to maintain reasonable incident detection procedures to disincentivize willful blindness. 

Allow the reportable categories to be updated without new legislation: Cal OES should be authorized to designate additional categories of critical safety incident by regulation. The flexibility that regulatory updating provides is crucial. As the Hugging Face incident showed, AI capabilities can evolve rapidly in ways that are difficult to predict in advance. It is therefore important for California to be able to update the incident reporting regime nimbly to capture unforeseen categories of concerning incidents. 

Attach the reporting duty to the developer rather than the model: SB 53 requires reports only for incidents involving a “frontier model,” i.e., a model trained on more than  10²⁶ FLOP. Publicly available data does not confirm whether or not the “Highly Persistent Internal Model” primarily responsible for the Hugging Face breach was trained on less than that amount of compute. Regardless of whether that particular model was a “frontier model” or not, increases in algorithmic efficiency will, in all likelihood, produce sub-10²⁶ models with capabilities exceeding those of the current frontier in the near future. To prevent SB 53’s incident reporting requirements from rapidly becoming obsolete, frontier developers should be required to report qualifying incidents involving any of their foundation models, whether released, internal, or still in training. 

Add reporting for near misses and other developer-identified relevant events: Developers should also report near misses and any other incident that, in their opinion, bears materially on public safety, public health, or national security. This broad catch-all reaches worrying incidents that objective criteria would miss, and because it is explicitly tied to what the developer itself believes to be material, its breadth does not create the legal uncertainty that a subjective standard otherwise might.

Give the state the power to investigate incidents: Incident reports only matter if they produce some kind of response, and in order to respond wisely to AI incidents, California’s government will need to be able to investigate. While existing law provides general investigative authorities that the California Department of Justice and other agencies can utilize for AI incident investigation, additional purpose-built authorities are needed. Authorize Cal OES to compel records and testimony, direct a developer to investigate specified questions, examine the systems involved with the aid of retained outside experts, and publish findings. Require developers to preserve evidence of an incident for five years.

Replace internal-use summaries with periodic risk reports: While responding to incidents is crucially important, California’s government should also receive important information about developing risks even when no incident has yet occurred. Require large frontier developers to report quarterly on risks within each catastrophic-risk category and from automated AI research and development, including risks from internal deployment and non-public access programs, and to include any determination that a model has reached or is likely to reach a capability that could pose a catastrophic risk. They should also update Cal OES within 24 hours when a risk substantially exceeds the level last reported. 

Let the government share what it learns: Authorize Cal OES to share incident reports and periodic risk reports with the federal government and with agencies of other states under equivalent confidentiality protections, to issue confidential alerts to developers exposed to the same vulnerability, and to publish anonymized findings on both a time-sensitive and annual basis.