Don’t Let AI Developers Hire Their Own Referees

Introduction

A growing chorus of scholars and policymakers favors letting private organizations—rather than a government regulator—govern frontier AI. In the leading family of proposals, the state sets the safety outcomes it wants and licenses independent verification organizations (IVOs) that compete to certify developers against those outcomes. Gillian Hadfield has developed the idea as “regulatory markets,” in which AI developers must pay for oversight from private regulators that governments license and hold accountable for safety standards. Dean Ball, who likens the arrangement to bank supervision, has argued for a version he calls “private governance,” which a nonprofit named Fathom has converted into model legislation. The rationale is that legislators and agencies are poorly positioned to write good safety rules for frontier AI: they understand these systems less well than the labs building them, and rules fixed in advance cannot keep pace as the technology changes. Private verifiers, meanwhile, are closer to the technology than any agency and are disciplined by competition, so they can set better technical standards and keep them current.

The model is no longer hypothetical. The bipartisan FRONTIER Act, introduced in the House in July as the successor to the Great American AI Act discussion draft, would require the largest frontier developers to retain licensed IVOs that audit their risk-management efforts and report to federal overseers. California’s SB 813, backed by Fathom, would have let developers earn a shield from tort liability if they met standards set by a private organization accredited by the state attorney general. It failed this session, but similar proposals are likely to return. Virginia has directed a state commission to study the IVO model for AI regulation. And Connecticut has gone furthest: its omnibus AI law enacted this spring creates a multiyear pilot under which the state consumer-protection department may approve up to five IVOs, whose certification would help companies in court without entirely shielding them from liability.

Unfortunately, as currently structured, IVO-based governance has a key design flaw. Under the regulatory frameworks mentioned above, AI developers would typically select and pay the organizations that certify them, giving IVOs a financial incentive that might clash with high safety standards. This essay will explain how such incentives can interfere with good governance and outline an alternative model of regulation that builds in the right incentives through mandatory insurance.

A Thousand AI Constitutions

Abstract

Today, each AI lab has its own model spec, or constitution. These documents define the values that the labs intend their AIs to have, and the documents are used in post-training to instill those values. This paper argues that the current approach is wrong. Rather than a single constitution, reflecting a single set of moral values, each frontier AI lab should create many different kinds of AIs based on many different constitutions reflecting many sets of values. We give four arguments for constitutional diversification. Diversification mitigates risk, increases political legitimacy, unlocks emergent value, and avoids value lock-in.

The NDAA: A Key Vehicle for AI Governance

Summary

Introduction

The National Defense Authorization Act has quietly become one of Congress’s most powerful tools for shaping AI policy, and the FY26 NDAA featured many key AI provisions. This commentary compiles all the major AI provisions from the FY26 NDAA and analyzes the most significant language in detail. With some of the initial deadlines imposed by the FY26 NDAA now having passed—and with negotiations around the FY27 NDAA underway—it’s useful to take stock of the potential and pitfalls of these provisions.

The NDAA is not just restricted to the nuts and bolts of defense operations. It has also been used to achieve broader policy goals, sometimes by limiting the executive branch’s actions. For example, one of the most important and successful nonproliferation programs in history, the Nunn-Lugar Cooperative Threat Reduction Program, was originally proposed as an amendment to the NDAA and was subsequently expanded through the NDAA.[ref 2] More recently, the FY19 NDAA effectively banned the government from using certain Chinese telecommunications companies such as Huawei and ZTE; likewise, the FY26 NDAA bans certain foreign AI products like DeepSeek.

As the government increasingly prioritizes AI use in warfighting and military operations, the NDAA has a key role to play in shaping AI policy.

AI Governance in the FY26 NDAA

Notable Provisions

Several provisions stand out as particularly important for the government’s broader interest in overseeing and fostering the responsible development of secure AI systems: 

Section 1535: Artificial Intelligence Futures Steering Committee

Section 1535 requires DOD to create an Artificial Intelligence Futures Steering Committee (Steering Committee) to prepare DOD for advanced AI and AGI. The Steering Committee will be co-chaired by the Deputy Secretary of Defense and Vice Chairman of the Joint Chiefs of Staff (VCJCS). It will primarily be composed of principal deputies of the military services, relevant under secretaries (e.g., the Under Secretary of Defense for Research and Engineering (USD(R&E))), and others responsible for AI (e.g., the Chief Digital and AI Officer (CDAO)).[ref 3]

By January 31, 2027, the Steering Committee must submit a report to Congress covering what can be described as two main focus areas. First, the committee must help prepare DOD for advanced AI and AGI by creating: 

  1. A proactive policy for the evaluation, adoption, governance, and risk mitigation of advanced AI systems, including systems that approach or achieve AGI. 
  2. An analysis of the forecasted trajectory of advanced AI models and enabling technologies that could lead to AGI such as AI agents, neuromorphic computing, cognitive science applications, infrastructure needs, new microelectronics, etc.
  3. An analysis of the potential operational effects of integrating advanced AI or AGI into DOD networks and systems from a technical, doctrinal, training, and resourcing perspective to better understand effects on operational commands. 
  4. A strategy for the risk-informed adoption, governance, and oversight of advanced AI and AGI including ethical, policy, and technical guardrails to maintain appropriate human decision-making and prevent misuse. 

The second focus area is U.S. adversaries. Though not specifically named, the People’s Republic of China (PRC), which is actively pursuing AI capabilities that rival those of the United States, is likely the primary focus. The committee must assess the possible technological, operational, and doctrinal trajectories of U.S. adversaries with respect to AI capabilities, including the pursuit of AGI. Additionally, the committee must analyze the threat landscape associated with the use of advanced AI and AGI and develop options to counter these threats. 

Within the Pentagon’s sprawling bureaucracy, there’s often fierce competition between different programs and priorities for funding and attention from leadership. In this sense, the Steering Committee could be a valuable forcing function for the department to prepare for advanced AI, reinforced by the requirement to report its findings to Congress by early 2027. There’s precedent for DOD using these sorts of committees as a way to spur action on issues such as software modernization and autonomous systems.

However, such committees sometimes serve more as a signaling mechanism for Congress than as a catalyst for serious action. Unless chairs or members of the committee invest their time and professional capital to drive it forward, it can easily devolve into a box-checking exercise. While the Steering Committee’s substantive mandate is broad, its required procedural actions, as set by Congress, are fairly minimal: meet at least once every three months, and submit a report on its findings to the relevant congressional committees by January 31, 2027. That means that depending on when the committee is actually established and how quickly it first convenes, it may meet only three or four times before its report is due. 

On top of that, the NDAA provision does not allocate any dedicated staff or budget for the Steering Committee. Any resources must be drawn from existing reserves, which could further limit its capacity. Given those constraints and their already-full plates, the Steering Committee’s principals might be tempted to delegate their roles and responsibilities down the chain of command to other, typically less-empowered subordinates, whose remit might be narrower—i.e., drafting a report that satisfies Congress’s requirements while potentially tabling thornier policy disagreements or implementation details for later.

Congress should remain attuned to these possible failure modes and use its oversight power to solicit information about the Steering Committee and its progress, in the hopes of helping it gain and maintain momentum. There are some encouraging signs on this front. In March, Senator Jim Banks sent a letter to Secretary Hegseth requesting a staff-level briefing within 60 days to discuss DOD’s plans for the Steering Committee. The letter suggested areas of focus with respect to U.S.-PRC AI competition. Even just one or a few members of Congress taking specific, sustained interest in the Steering Committee could keep it high enough on DOD’s long list of priorities to increase its odds of success. 

Congressional oversight can be particularly valuable in two ways. First, it can keep pressure on the committee if it fails to meet the report submission deadline of January 31. Second, and perhaps more importantly, Congress can help ensure that the Steering Committee doesn’t waste the 11 months between its reporting deadline and its termination date of December 31, 2027. 

While the report is the Steering Committee’s most tangible required deliverable, Congress provided that the committee will continue to exist for nearly a year beyond the report submission deadline. This time would allow the committee to refine or update its policies and to work on implementing and disseminating the findings throughout DOD. Because the Steering Committee lacks deliverables or other measurable benchmarks throughout most of 2027, it’ll likely be incumbent on Congress to use tools like letters, hearings, and requests for briefings to push forward that updating and implementation work. These efforts could ultimately have a much greater impact on DOD operations in the long term than just the drafting of the report itself.

As of June 30, 2026, no public materials indicate whether the Steering Committee was established by the April 1 statutory deadline, or whether it has held its first meeting. That’s not necessarily cause for concern, as DOD is not required by the NDAA to report those actions to Congress or the public. But it does make it harder to predict which of these paths the AI Futures Steering Committee will ultimately follow. Overall, this provision could pay dividends by prompting DOD to proactively prepare for major threats and opportunities raised by AGI—planning that might otherwise get neglected—though its success is far from assured. 

Key Dates

Section 1533: AI Model Assessment and Oversight

Section 1533 instructs DOD to create a Cross-Functional Team (Team) for AI “model assessment and oversight.” The Team must develop a standardized assessment framework for AI models currently used by DOD, as well as guidelines to facilitate procurement of future models. The Team is led by the CDAO and composed of other DOD technology leaders, such as CIOs, CAIOs of the combatant commands, service acquisition executives, and USD(R&E). The Team must: 

This provision allows DOD to retain a lot of discretion over how it evaluates current and future AI models. Congress has mandated that DOD establish a framework and protocols, but didn’t set substantive thresholds for performance. That’s understandable to some degree, given the risk of setting standards via legislation, which might quickly become outdated and then prove difficult to adjust. And it’s similar to the approach that states like California and New York have taken in enacting frontier AI transparency reporting requirements. But some key requirements in the provision, such as the creation of “governance structures” and assessing “ultimate use-case-based risk,” use terms that are undefined and open to interpretation, and could have benefited from a bit more congressional guidance about the elements that should at least be considered or addressed.

That vagueness, combined with the long timelines the provision establishes, could make it hard for Congress to assess the Team’s progress. Congress notably gave the Team an extended timeline to develop its model assessments and oversight, which may be in tension with the pace of AI progress. The standardized assessment framework isn’t due until June 2027—a year and a half after enactment—and no actual assessments of DOD’s major AI systems are required until January 2028. Meanwhile, new frontier AI models are released many times a year. 

To be sure, the Team’s task is difficult. And Congress sometimes errs by giving agencies unrealistically short deadlines. But a failure to keep up with the pace of AI development risks undermining the Team’s purpose. To frame that risk, consider the events that have transpired since the FY26 NDAA passed six months ago. First, there was the blow-up over contract terms between the Pentagon and Anthropic in February. More recently, the June 5 National Security Presidential Memorandum (NSPM) 11 ordered Secretary Hegseth, ODNI, and IC elements to “review and update procurement processes to ensure the rapid onboarding of the most advanced AI models from multiple vendors” within 120 days. It’s unclear how or whether this review will be coordinated with the procurement guidelines that the Team is tasked with developing on its longer timeframe.

Here, again, Congress can deploy its oversight tools to steer DOD in the direction of consistent and streamlined guidelines for AI procurement. It should aim to ensure that standards are applied uniformly and transparently, not reactively, to AI developers. Helpfully, this provision requires DOD to provide a briefing to congressional defense committees within 30 days of hitting significant statutorily prescribed milestones, starting with its establishment of the Team on or before June 1, 2026. That offers a natural opening for Congress to probe the Team’s trajectory, and potentially to spur a course correction if needed. Congress might consider incorporating that sort of regular briefing requirement into future AI-related NDAA provisions; it’s particularly beneficial in this area due to rapid and sometimes unexpected jumps in capabilities and risks, and might also have been helpful for similar initiatives like the Steering Committee discussed above.

Finally, it’s worth a closer look at the provision’s definition of “major [AI] system”—one of only a few terms that the provision does actually define—buried near the end of the provision. That definition limits coverage to systems used annually by at least 500 users within DOD, and excludes systems used solely for research, development, testing, or evaluation that have not been deployed for operational use. Elsewhere, the provision specifies that DOD must assess all major AI systems using the standardized assessment framework, leaving it somewhat unclear when or to what extent that framework also governs assessment of other AI models used by DOD. In other words, for models used by less than 500 employees per year, or those involved only in R&D, how will DOD assess performance, security, and “compliance with ethical principles”? 

While this sort of line-drawing exercise is almost always difficult but necessary for administrability, in this instance the exclusions arguably represent the frontier of DOD’s own AI development and deployment in what could end up being the highest-stakes and hardest-to-monitor situations. At minimum, it’s plausible that some of the most powerful systems, deployed in potentially highly consequential cases, might be available to only a small number of users. Congress should ask DOD how it plans to assess AI systems that fall into those categories and potentially require the development of standards for such systems in future legislation.

Key Dates

Section 1534: Digital Sandbox Environments for AI

Section 1534 requires the CDAO to create a task force to promote AI sandbox environments supporting “experimentation, training, familiarization, and development.” The task force should “identify, coordinate, and advance” DOD efforts to develop and deploy AI sandboxes, with an eye toward accelerating AI adoption across the department. The provision defines an “[AI] sandbox environment” as a “secure, isolated computing environment that enables users with varying levels of technical proficiency to access [AI] tools, models, and capabilities for the purposes of experimentation, training, testing, and development without affecting operational systems or requiring specialized technical knowledge to operate.” The provision requires that the task force be established by April 1, 2026, and that the CDAO provide a briefing to congressional defense committees by August 1 on the task force’s goals and objectives.

One noteworthy aspect of this provision is the emphasis that Congress has placed on using sandboxes to facilitate training and familiarization with AI by DOD employees—“from personnel with little technical proficiency to personnel with expert technical proficiency.” Congress should be commended for devoting at least as much attention to that purpose as to how sandboxes are used to develop and test AI tools and models, which is often the main or even exclusive focus of sandboxing. In an organization as large and varied as DOD—and in which the stakes are matters of national security—giving employees a dedicated environment in which to try (and fail) so as to ultimately gain a level of comfort using novel and quickly evolving AI systems is critical to the widespread adoption that Congress is after.

One area where both DOD and Congress might focus some more attention during the required briefing is how the task force can facilitate a pipeline between successful AI development that occurs in sandboxes and the actual implementation of those systems, tools, or methods in the real world of DOD operations. That’s a topic that the provision as written doesn’t address as squarely, but it’ll be key to ensuring that DOD can fully capitalize on its investment in AI sandbox environments. DOD can be a process-heavy place at times; the task force will need to plan for how to judge when AI experiments are ready to graduate from sandboxes, and to efficiently move those successful innovations from sandboxes to the rest of the department.

Key Dates

Section 1513: Physical and Cybersecurity Procurement Requirements for Artificial Intelligence Systems

Section 1513 requires DOD, in collaboration with industry and academia, to develop a framework for the implementation of cybersecurity and physical security standards and best practices for AI systems, “to mitigate risks to [DOD] from the use of such technologies.” The framework must cover enumerated concerns like insider threats, data poisoning, and adversarial tampering. The provision also instructs that the framework must be “risk-based,” drawing on existing reference documents, including NIST’s SP 800 series, and augmenting existing cybersecurity frameworks, including DOD CMMC. 

To implement the best practices developed under the framework, DOD must amend the Defense Federal Acquisition Regulation Supplement (DFARS) “or take other similar action” ensuring that those practices apply to contractors who engage in AI development, deployment, storage, or hosting. In carrying out that function, DOD must weigh the costs and benefits of imposing security requirements on contractors—and specifically, the costs of “slowing down” AI development and deployment against “the benefits of mitigating national security risks and potential security risks” to DOD.

While this provision is expressly attuned to the potential costs of slowing down AI development through unduly onerous security requirements, it’s at least equally concerned with mitigating the risks to DOD—and national security more generally—that AI systems can pose. It will be worth monitoring how the framework approaches that statutorily required balancing, not least because of how it contrasts with the January 9 AI Strategy memo issued by Secretary Hegseth, which seemingly prized speed above all else. 

Lines from that memo, like “speed wins,” and “We must accept that the risks of not moving fast enough outweigh the risks of imperfect alignment,” offer a preview of where DOD seems most likely to come down on these issues. They also suggest that Congress may have to be dogged in reviewing a required June status update and pursuing other oversight measures to confirm that the statutorily mandated cost-benefit analysis is sufficiently rigorous, with real attention to serious risks Congress mentioned, such as adversarial tampering.

Key Date

Section 1061: Notification of Waivers under DOD Directive 3000.09 

Section 1061 requires DOD to notify congressional defense committees when it has waived DOD Directive 3000.09 (DoDD 3000.09) relating to the use of autonomous weapon systems (AWS).[ref 6] The notification must be in writing and transmitted to the relevant committees within 30 days of when the waiver was issued. The notification also must be unclassified and must include the rationale for the waiver, a description of the weapons system or technology covered by the waiver, and the anticipated duration of the waiver. DOD may include a classified annex to the waiver, as necessary.

DoDD 3000.09 states that “[a]utonomous and semi-autonomous weapons will be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force” (emphasis added). As Kelley Sayler of the Congressional Research Service has noted, that does not mean that “manual human ‘control’” of the system is required, but rather mandates “broader human involvement in decisions about how, when, where, and why the weapon will be employed”—for example, “a human must assess the operational environment and decide to deploy the weapon, which can then operate autonomously.” 

As most relevant here, DoDD 3000.09 allows for DOD to skip the traditional review and approval process for AWS when there is an “urgent military need.” Typically, the Under Secretary of Defense for Policy (USD(P)), USD(R&E), and the VCJCS must approve a system before formal development, and then it must be approved again before being deployed in operations by the Under Secretary of Defense for Acquisition and Sustainment, USD(P), and VCJCS.[ref 7] DoDD 3000.09 allows any of these parties to request a waiver of the policy requirements per approval of the Deputy Secretary of Defense.

Section 1061 is the latest in a series of recent NDAA provisions through which Congress has sought greater insight into DoDD 3000.09, particularly whether and how it’s being applied or modified. In the NDAA for fiscal year 2024, Congress required that DOD provide a briefing to congressional defense committees within 30 days of making any changes to DoDD 3000.09, including a description of the change and an explanation of the reasons for it. In fiscal year 2025’s NDAA, Congress required DOD to submit annual reports to those committees through December 31, 2029, on its approval and deployment of lethal AWS under DoDD 3000.09, including any systems that received a waiver from the policy’s review requirement.

This is a prime example of Congress using the NDAA to iterate and build progressively on existing requirements as issues rise in salience—and the salience of DoDD 3000.09 has arguably never been greater. The directive featured prominently in the Pentagon’s dispute with Anthropic earlier this year. Furthermore, NSPM-11 issued by President Trump on June 5 orders Secretary Hegseth to update DoDD 3000.09 within 90 days, and to review it annually “to account for the rapidly evolving capabilities of AI systems” and “ensure the deliberate adoption of AI systems that respect the chain of command and operational authorities.”

In keeping with this progression, one valuable adjustment to Section 1061 that Congress might make would be an amendment that requires an update to the committees when the duration of a waiver is extended beyond the “anticipated” period previously notified, as well as regular updates for any waivers that DOD issues that don’t have a specified end date or timeframe. This would help to guard against overreliance on waivers that might be open-ended or persist for years without prompting congressional scrutiny. Otherwise, waivers issued in prior years might not necessarily show up in the annual reports required under the NDAA for fiscal year 2025.

Going further, Congress could consider whether to codify all or parts of DoDD 3000.09, potentially preserving DOD’s ability to waive or deviate from aspects of the policy when warranted to avoid restrictions that might prove too rigid or become quickly outdated. Both the House and Senate FY27 NDAA markups address DOD AWS policy, though with notable differences. While the final text of any AWS-policy provision in the FY27 NDAA may differ substantially from the markups, these initial versions shed some light on possible approaches. 

The House markup requires that DOD update its AWS policy, including DoDD 3000.09, within one year of enactment—significantly longer than the 90 days DOD has to update the directive under NSPM-11. But as compared to the NSPM, the House markup provides more detail on what an updated policy must include, not least “requirements to preserve existing human command responsibility for the use of force involving autonomous systems or artificial intelligence-enabled systems, including procedures to identify the human commanders or operators responsible for authorizing, supervising, and terminating such use of force.” The Senate markup goes much further still, prescribing an AWS policy and governance regime for DOD in significantly greater detail, with an even more defined substantive floor. And while the Senate markup in multiple places incorporates DoDD 3000.09’s familiar standard of “appropriate levels of human judgment,” it does not directly address the directive’s existing waiver process, leaving it unclear whether that aspect of DoDD 3000.09 would pass muster and thus survive the substantive standards established by this provision.

If Congress opts for a more prescriptive approach, it could consider adding a sunset clause to hedge against the risks of excessive rigidity or obsolescence. A short initial timeline of 1–2 years would prompt Congress to revisit and adjust as needed, providing a short feedback loop for any DOD operational concerns or issues that emerge. 

The Road Ahead: What to Watch for in 2026 and 2027

The FY26 NDAA showed how the annual defense bill can be one of—or even the—primary vehicle for the governance and oversight of defense-relevant AI decisions. Congress can use it to spur prioritization and adoption (Steering Committee and sandboxes), mandate the development of standards and assessments (AI model oversight), prompt consideration and safeguarding against security risks (cybersecurity procurement requirements), and gather information about how the department is using AI (autonomous weapons waivers and various briefing requirements in other provisions). 

Throughout the remainder of 2026 and beyond, it’s worth continuing to monitor updates to key provisions via congressional briefings and other potential disclosures, especially regarding autonomous weapons waivers and the implementation of an AI physical and cybersecurity procurement framework and AI model assessment and oversight. At least one of these initiatives, the AI physical and cybersecurity procurement framework, expressly requires that DOD seek input from groups like industry and academia. Experts should look for opportunities to engage through requests for information or other formats. Congress also has a significant role to play in ensuring that implementation proceeds responsibly and on schedule, using oversight tools like letters, briefing requests, and hearings to supplement the reporting requirements baked into some, but not all, of the key provisions.

As negotiations for the FY27 NDAA ramp up, we can expect numerous AI initiatives to be considered and ultimately included—perhaps even more than last year, since other legislative vehicles will likely be few and far between in this midterm election year. The current House and Senate FY27 NDAA markups include provisions on AI incident and vulnerability reporting within DOD, using AI agents at scale and speed, and promoting competition in AI procurement. The FY27 NDAA could also serve as the vehicle for another attempt at federal preemption of state AI laws, which was dropped shortly before last year’s bill was passed. 

In all of these, Congress should learn from last year’s NDAA. It should craft implementation timelines for DOD that provide space for careful consideration but are not overly long relative to the rapid rate of technological development and diffusion. And it should think about where to build in briefing and other reporting requirements to fill in its knowledge gaps regarding implementation, while being sensitive to the demands they impose on personnel’s time. Doing so helps Congress not only ensure that last year’s initiatives are proceeding according to plan, but also provides valuable insight about unexpected challenges or shortcomings that can inform the coming year’s bill.

Whistleblower Protections in SB 53: Strengths, Limitations, and Open Questions

Executive Summary

SB 53 greatly improves the AI safety legislative landscape in California, but achieves only partial success in its whistleblowing protections. Its major success is that it significantly increases the number of safety-relevant issues that can be reported to authorities. In particular, employees can now blow the whistle when they believe frontier developers have not reported critical safety incidents or where large frontier developers have made materially false or misleading statements about, or are not complying with, their AI framework. Employees “responsible for assessing, managing, or addressing risk of critical safety incidents” will also be explicitly protected when blowing the whistle about catastrophic risks.  SB 53 also requires large frontier developers to provide an anonymous internal reporting channel that a select group of employees can use to report catastrophic risks and that all employees can most likely[ref 1] use to report a violation of the law (including of SB 53 itself).

Looking more deeply at the provisions of SB 53, however, we see a more mixed picture. An ideal AI whistleblower law would ensure that all reasonable risks could be reported; that all individuals capable of spotting such risks could report them without fear of retaliation; that any relevant individual could support government  investigations without fear of retaliation; that authorities are equipped to investigate disclosures; and that real consequences exist for violations. SB 53 achieves each of these partially. 

First, the responsibility for disclosing catastrophic risks is not placed onto companies, but instead on an unclearly-defined subset of employees. SB 53 does not require companies to report a specific and substantial danger to the public health or safety resulting from a catastrophic risk to the Office of Emergency Services (OES). Nor do evaluation providers receive any protection from retaliation, either for making reports or for participating in government investigations. 

Second, while the definition of catastrophic risk is relatively broad and does not require a clear violation of the law, it is limited to specific types of catastrophic risk and harms only above a high threshold: fifty deaths or a billion dollars in damages.

Third, although SB 53 creates incentives for companies and insiders to report harm caused by AI, it is unclear what consequences these reports will entail. Company incident reports are sent to the OES, where they might not be accessible to the public, and whistleblower reports of catastrophic risk to California’s Attorney General (AG) or a federal authority. The OES lacks clear powers to compel companies to change their behavior, the AG has no clear mandate to intervene in the absence of  a legal violation, and no federal authority has established jurisdiction over such risks. 

Lastly, SB 53 does not adjust the penalties for violating whistleblower protections in California, which are meagre at $10,000 per violation — a far cry from the $1m violation cap for other SB 53 violations, and extremely little in the context of large frontier developers, defined as having gross revenue of over $500,000,000. 

The next step is to ensure SB 53’s faithful implementation. State agencies must build the necessary expertise: the OES and the AG need access to specialised expertise to evaluate and respond to critical safety incidents and catastrophic AI risks. These agencies should collaborate closely to develop a comprehensive understanding of emerging threats and assess potential legal violations.

Background

SB 53’s whistleblower provisions are legally entwined with California’s pre-existing whistleblower framework, which they extend, and SB 53’s core transparency requirements, for which they serve as an enforcement mechanism. 

CA Labor Code § 1102.5 is California’s general whistleblowing law, covering all industries and employees. It prohibits employers from retaliating against employees who disclose information, which they have reasonable cause to believe to be a violation of the law, to a government or law enforcement agency, or to a person with authority over the employee. These protections cover a wide range of potential violations.  Nevertheless, California’s protections are narrower than some other states, such as New York, as they (i) only cover employees and do not extend to contractors and (ii) only protect disclosures of violations of the law, and do not protect employees from retaliation for disclosing a substantial and specific risk to public health and safety. This is particularly significant in the context of AI as frontier AI companies frequently rely on contractors for safety-relevant roles such as red-teaming and safety evaluations[ref 2] and novel AI risks often do not constitute clear legal violations, making the ability to report risks to public health and safety essential for early intervention. 

SB 53, or The Transparency in Frontier Artificial Intelligence Act,  signed into law in September 2025, creates a transparency framework for the most powerful AI systems. The Act applies to “frontier developers”, defined as persons who have trained or initiated the training of AI models using extraordinary computing power (greater than 1026 FLOPs) and imposes heightened obligations on “large frontier developers”, defined as  frontier developers with over five hundred million dollars ($500,000,000) in annual revenue. As of 26th January 2026, the only publicly known frontier developers are xAI and OpenAI, while the only large frontier developer is OpenAI.[ref 3] Beyond whistleblower protections, outlined below, SB 53’s main contribution is introducing transparency requirements, spanning four key areas: i) Frontier AI Framework Requirements: Large frontier developers must publish annually-reviewed protocols for managing catastrophic risk; ii) Transparency Reporting Requirements: Developers must publish summary reports of features and risks, similar to existing model cards and system cards, before deploying new or substantially modified models; iii) Government Reporting Requirements: Frontier developers must report critical safety incidents to the OES within 15 days (or 24 hours if posing imminent risk of death or serious injury), with large developers also submitting quarterly risk assessment summaries; iv) Prohibition on materially false statements: Large frontier developers are prohibited from making materially false or misleading statements about catastrophic risks from their models or their management of such risks. 

Crucially, these requirements are legally binding. As such, companies that fail to comply are in violation of California Law and hence whistleblower protections apply to any employee who reports them and are a critical oversight mechanism. 

Policy Analysis

SB 53 extends pre-existing whistleblower protections to cover reporting catastrophic risks as well as legal violations, creates mandatory internal reporting channels, and enables employees to report critical compliance failures directly to authorities. Our analysis proceeds through four dimensions: personal scope, material scope, remedies, and channels. For each aspect of whistleblowing law, we discuss the advantages and limitations of SB 53’s provisions. 

Personal Scope

Personal scope defines who can make a whistleblowing disclosure. Whistleblowing law is part of the California Labor Code and hence only governs employees whose employment contract is under California law. SB 53 creates a tiered protection structure: all employees receive protection when reporting violations of SB 53’s requirements; and a subset, ‘covered employees’, defined as those responsible for assessing, managing, or addressing risk of critical safety incidents, receive protection for reporting catastrophic risks that do not include a legal violation. The scope of ‘covered employee’ remains ambiguous, creating uncertainty about who falls into this protected group. Nevertheless, the breadth of the language suggests that most employees whose work relates to AI safety in some way are likely “covered”.

Advantages

Limitations

Material Scope

Material Scope defines the subject matter that can form the content of a whistleblowing disclosure. SB 53’s key contribution is extending  California’s whistleblower protections beyond legal violations by protecting covered employees from retaliation for reporting “a specific and substantial danger to the public health or safety resulting from a catastrophic risk” with “reasonable cause to believe” such danger exists (§ 1107.1(1)). To understand this scope, two definitions and their thresholds are essential: “critical safety incident” and “catastrophic risk.”

“Critical safety incident” means any of the following:

  1. Unauthorized access to, modification of, or exfiltration of the model weights of a foundation model that results in death, bodily injury, or damage to, or loss of, property.
  2. Harm resulting from the materialization of a catastrophic risk.
  3. Loss of control of a foundation model causing death or bodily injury.
  4. A foundation model that uses deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside of the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased catastrophic risk.

“Catastrophic risk” means a foreseeable and material risk that a frontier developer’s development, storage, use, or deployment of a foundation model will materially contribute to the death of, or serious injury to, more than 50 people or more than one billion dollars ($1,000,000,000) in damage to, or loss of, property arising from a single incident involving a foundation model doing any of the following:

  1. Providing expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
  2. Engaging in conduct with no meaningful human oversight, intervention, or supervision that is either a cyberattack or, if committed by a human, would constitute the crime of murder, assault, extortion, or theft, including theft by false pretense.
  3. Evading the control of its frontier developer or user.

The Act expressly excludes risks from the category of catastrophic risks: information generated by a model if it is already publicly available, lawful federal government activity, and situations where the model causes harm in combination with other software, but AI did not materially contribute to the harm. Thus ‘catastrophic risk’ in the Act only refers to CBRN and loss of control risks.

Advantages 

Limitations

Remedies

SB 53 applies existing Labor Code remedies to its whistleblower provisions. This section examines the protections available to whistleblowers and deterrents against retaliation.

Advantages

Limitations

Channels

Whistleblowing channels are the individuals, agencies, or offices to which a whistleblower disclosure can be made. SB 53 protects covered employees if they whistleblow “to the AG, a federal authority, a person with authority over the covered employee, or another covered employee who has authority to investigate, discover, or correct the reported issue” (§ 1107.1(a)). Additionally, SB 53 requires the creation of “a reasonable internal process through which a covered employee may anonymously disclose information to the large frontier developer if the covered employee believes in good faith that the information indicates that the large frontier developer’s activities present a specific and substantial danger to the public health or safety resulting from a catastrophic risk” or a violation of SB 53’s transparency obligations. Under the California Labor Code, any employee is protected if they whistleblow to “a government or law enforcement agency, to a person with authority over the employee or another employee who has the authority to investigate, discover, or correct the violation or noncompliance,” or “any public body conducting an investigation, hearing, or inquiry” (§ 1102.5(b)). California’s Labor Code also provides a whistleblower hotline, operated by the AG, intended to receive calls from whistleblowers. Following a report made to the hotline, the  AG is directed to refer calls received on the hotline to the appropriate government authority for review and possible investigation. 

Advantages

Limitations

Advice and Consent for Major Governmental AI Deployments

On March 26, Judge Rita Lin of the U.S. District Court for the Northern District of California granted a preliminary injunction for Anthropic in its ongoing dispute with the Department of Defense over its designation of Anthropic as a supply chain risk. The Anthropic affair has intensified a debate over how law should constrain the executive branch’s use of artificial intelligence (AI). Much of this debate centers on the types of substantive rules at the heart of the Anthropic dispute, such as: Should large-language models (LLMs) be used for lethal autonomous weapons? What rules should govern the use of AI to process intelligence? What exactly do “any lawful use” provisions allow, and who decides?

While Congress should resolve many of these matters, its ability to prospectively regulate governmental AI systems is limited. Developing and deploying AI systems in government settings requires myriad choices, many of which could impact safety, security, and constitutional values. Congress cannot easily anticipate all of these choices, much less decide on rules for them in advance. This creates a dilemma: A complete legislative specification of these design choices will remain impossible, but reliance on the good faith of the executive branch to govern its own AI deployments responsibly is equally unwise.

Additional oversight mechanisms are therefore necessary. This article proposes a new approach: Congress should require the executive branch to secure affirmative congressional approval before deploying AI in certain high-risk domains. Here’s how this could work. Congress would first designate certain governmental domains as “Protected Use Cases”: areas of use where, in its judgment, additional congressional oversight is warranted. Congress would then enact a default rule prohibiting the use of (specific types of) AI for Protected Use Cases.

To overcome this prohibition, the president would be required to submit to Congress a detailed proposal for deploying AI in a Protected Use Case. This would include, for example, a particularized description of the AI system, planned guardrails, authorized and prohibited use cases, authorized users, oversight affordances, and permitted modifications. After back-and-forth negotiations between the branches, Congress would vote on ordinary legislation to provide a specific exception for the proposed deployment, as amended by Congress during the legislative process. Once the bill becomes law, the specific deployment would be approved, but the background prohibition on Protected Use Cases would remain for other, nonapproved AI systems.

The resulting dynamics would loosely mirror the constitutional advice-and-consent process for principal officers, granting Congress an additional mechanism for subjecting powerful governmental decision-makers to public oversight and democratic legitimation of governmental AI systems, without the need to rely entirely on prospective legislation.

Scalable Subordinates

AI systems are increasingly indispensable tools of governance. Many of the central tasks of public administration—collecting and analyzing information, investigating possible offenses, and coordinating the organs of state toward substantive policy goals—seem ripe for augmentation, acceleration, and automation by AI systems.

But with this potential also comes great peril. The executive branch is checked in significant part by its human workforce. Soldiers and civil servants have diverse values and interests that typically diverge from the president’s to some degree. They have an independent obligation to obey the law and uphold the Constitution, while lacking many of the legal protections afforded to the commander in chief. This gives them powerful incentives to resist executive branch abuses.

Replacing human government workers with AI systems could unsettle these dynamics. Without specific requirements to the contrary, governmental AI systems could be built to be unwaveringly obedient. The systems would not fear criminal liability, worry about their post-government reputation, or suffer crises of conscience. They could act in unison at superhuman speeds, meticulously covering their tracks and frustrating attempts to hold them or their principals accountable.

Individual human employees or officers are also sometimes wicked, of course. But the downside risk from any given individual is limited. AI systems, by contrast, are highly scalable: Eventually, an AI system could perform the work of entire bureaucracies.

This structural shift suggests a case for recalibration of Congress’s checking function. One of Congress’s most significant powers is the constitutional requirement that the Senate confirms the executive branch’s principal officers. Limiting this requirement to principal officers was a prudent response to the realities of human administration. If the Senate needed to consent to the appointment of every officer or employee, it would have little time for other business. It therefore made sense to focus senatorial scrutiny on the top of the organizational chart. Indeed, the number of Senate-confirmed positions is arguably already too large.

But if a small number of executive branch AI systems could perform nearly all the work of entire departments, the design of those systems would be at least as important as the selection of a department head. This suggests that congressional involvement in the selection and design of those systems could be similarly appropriate.

Red Lines and Gray Areas

A congressional approval requirement makes sense for another reason: It would be unwise for Congress to rely solely on prospective legislation to govern the design, deployment, and use of executive branch AI.

The prospect of a highly automated executive branch has previously motivated me to argue that governmental AI systems should be designed to follow the law. Bright-line rules governing the procurement, deployment, behavior, and instruction of governmental AI systems will be crucial to guarding against their misuse.

But as any law student knows, it is impossible to identify and agree on rules for every imaginable scenario an actor might face. AI systems themselves might reduce the costs of the legislative process, but the complete specification of desirable behavior will always elude us. This presents a significant limitation of our prior proposal to require that governmental AI systems be “law-following AIs.” The law will always have gaps, and clever AIs will be good at finding them. Law-following AI, on its own, cannot prevent AI systems from exploiting such loopholes, because they remain legal by definition.

Some recent trends in frontier AI development vividly illustrate this point. Frontier AI companies rely on documents called “model specifications” or “constitutions” to shape the behavioral propensities of their AI systems. These documents contain bright-line rules, such as “never [p]rovide serious uplift to those seeking to create biological, chemical, nuclear, or radiological weapons with the potential for mass casualties.” But the documents also rely on a sort of Aristotelian virtue ethics that cannot be formalized as legal requirements. In other words, it may be important that executive branch AI systems have good “character” that dictates how they should navigate moral or legal gray areas. While a general legislative definition of “good character” would be unwise and difficult to pin down, assessment of whether a particular AI system has good character mirrors the familiar confirmation hearing process, where holistic assessment of a nominee’s past conduct and disposition is fair game.

The impossibility of a completely specified code for AI behavior is an instance of a more general issue: Safe and socially beneficial AI behavior is the product of a large number of design choices. Congress could not—and should not attempt to—prospectively dictate how the executive branch must make these design choices. Similarly, evaluation of AI systems is far from a mechanistic science: Proper testing involves making a large number of context-specific judgment calls that would elude legislative prespecification. However, assessing claims about specific AI systems is significantly easier. A congressional approval regime would enable such flexible assessments and allow Congress to reject proposed deployments on the basis of considerations that could never have been operationalized in prospective legislation.

To enact this policy, Congress would first need to define the set of executive branch deployments that require specific congressional approval. These are the Protected Use Cases. Careful demarcation of the Protected Use Cases is critical: Too narrow a scope could enable executive-branch misfeasance, while overbreadth would hamper state capacity.

Congressional preapproval is most important for systems that will exercise certain functions that pose a significant risk to the constitutional order, and for which after-the-fact remedies are limited. Imagine, for example, an AI system that integrates existing sources of lawfully collected information across multiple agencies, including purchased data, open-source information from the internet, intelligence sources, public security camera footage, and information already reported to the government (such as mortgage applications). The AI system then identifies possible crimes, obtains warrants for further information, and decides whether to recommend them for prosecution.

It seems likely that such a system would identify thousands of previously unidentified crimes. Perhaps, in isolation, this could be a valuable way for the government to catch more criminals. But the opportunities for abuse are obvious. Absent technical safeguards, it would be easy for a malicious government to use such a system to scrutinize political opponents systematically while ignoring political allies. Given the difficulty of successfully arguing a defense of selective or vindictive prosecution, this could be a powerful tool for political repression.

A full enumeration of candidate Protected Use Cases is beyond the scope of this piece. However, uses that might warrant inclusion include criminal investigations, domestic intelligence-gathering and analysis, domestic military deployments, and prosecutions. While this would include a wide range of governmental functions, it would also leave many core functions unimpeded: The work of many federal agencies could be entirely exempted.

Congress would then establish by statute that AI systems could not be used in a Protected Use Case without explicit statutory authorization. This basic structure is similar to familiar statutes such as the Posse Comitatus Act, which prohibits the use of regular military forces for domestic law enforcement without express congressional or constitutional authorization.

Congress should then statutorily preapprove certain deployments within the Protected Use Cases, meaning additional congressional approval would not be necessary. At least two categories of preapproval seem prudent. The first set of preapprovals would be for classes of AI systems that seem unlikely to pose the severe risks to liberty that motivate this proposal. One candidate class would be so-called narrow AI systems. While there are serious risks associated with these technologies, their adoption seems unlikely to displace human civil servants to a destabilizing degree.

Congress should also statutorily preapprove testing and development of AI systems for Protected Use Cases. This statutory scheme depends on the executive’s ability to propose well-specified deployments to Congress for approval. Departments and agencies cannot do this unless they can develop and test AI systems for Protected Use Cases without prior congressional approval. Of course, it would be important to carefully define “testing and development” so that it could not be abused to circumvent the general preapproval requirement. However, this is a manageable legislative drafting task.

Any governmental deployment of AI in a Protected Use Case not covered by a preapproval provision would be prohibited by default. Federal employees who willfully violate this prohibition would be subject to civil and criminal penalties, as would vendors who willfully aid in illegal deployment. This enforcement mechanism would parallel (and supplement) the Antideficiency Act, which prohibits federal employees from expending federal resources in excess of statutory appropriations. The executive branch would therefore need congressional approval for deployments for Protected Use Cases not covered by statutory preapprovals. Here’s how I envision that process working. Congress would give the executive branch a generous budget to develop and test prototypes for Protected Use Cases. This generous funding would help entice vendors to participate in prototype development, notwithstanding the prospect of Congress rejecting a proposed deployment. And the executive branch, cognizant of the need for congressional approval, would conduct the development and testing phase to maximize the likelihood that Congress signs off. They would, for example, brief the relevant committees about development progress and seek input on which tests should be conducted.

When the time is right, the executive branch would submit to Congress a report for proposed deployment of an AI system for a Protected Use Case. Obviously, since the approval requirement and process are entirely statutory, Congress cannot bindingly dictate the scope of future congressional approvals. But the policy goals of this process would be best advanced if the proposed deployment were fairly specific. It should include, for example, a technical description of the system being deployed, such as the AI models that power it and any accompanying technical safeguards. It should also include the results from the testing process.

But the proposed deployment need not be limited to technical information. For example, Congress might establish accompanying rules regarding who is allowed to operate the system and the purposes for which it may be used. It might also insist on reporting requirements, which could possibly be automated by other AI systems. Indeed, the design space for a proposed deployment would be large and multidimensional: Congress could create technical, legal, informational, and organizational safeguards that are tailored to the risks the system poses. In particularly risky cases, they could, for example, make liberal use of sunset provisions, impose demanding reporting requirements, and provide that failure to comply with any of the required safeguards voids the approval entirely.

However, it is equally important for Congress to enable the continuous development of governmental technologies. To facilitate this, Congress could explicitly greenlight certain modifications to an approved deployment while explicitly prohibiting others. Changes to maintain core functionality, improve user interfaces, and bolster cybersecurity could be allowed expressly, while adaptation of the software for new, riskier use cases could be expressly prohibited. The scale of the deployment could also be carefully managed, with Congress expanding the permitted scope of deployment over time as the product matured and Congress gained more confidence in it.

After back and forth negotiations and modifications, Congress would authorize the proposed deployment—including required safeguards and permitted alterations—through ordinary legislation. Once signed into law (or enacted over a presidential veto), the proposed deployment would be authorized. Congress would then use its oversight functions (both those specifically mandated in the proposed deployment and its general oversight powers) to ensure that the government did not exceed the scope of its authorization.

Congress Should Do Its Job

Perhaps the most compelling objection to this proposal is that it tasks Congress with forming nuanced views about the safety and security of proposed AI deployments. At a time when Congress is less popular than ever and hobbled by gridlock, is trusting Congress with more responsibility for complex technical matters really wise? More pointedly: Wouldn’t the dysfunctional and highly polarized nature of the legislative process risk kneecapping state capacity?

These are all reasonable concerns. Yet it’s difficult to imagine how the technological transformation of the executive branch could be anything other than disastrous without more aggressive congressional oversight. The Madisonian vision of legislative ambition checking presidential ambition has not played out as the framers imagined, but it is nevertheless an indispensable part of our system.

Item-specific control over executive branch resourcing decisions may not be the norm, but it has precedent. Early Congresses “exercised fine-grained control over all funding decisions,” including in military matters, down to “the precise numbers of troops [and] their alotted [sic] daily rations.” Today, Congress is still intimately involved in major procurement decisions, particularly in defense. Congressional oversight of specific AI procurements could be similarly critical.

If we begin to rely on Congress to evaluate executive branch technology, we should provide it with the necessary resources. It is cliché to argue that Congress should revive (something like) the Office of Technology Assessment (OTA), the now-defunded agency that assessed the likely impacts of emerging technologies for Congress. But it remains true: Congress will need in-house expertise to help it understand the technologies the executive branch requests. While the Government Accountability Office and Congressional Research Service make admirable efforts to provide Congress with reliable information about AI and other emerging technologies, the imagined policy here will require a dedicated organization for AI assessment. Nor, for separation of powers reasons, would it be wise for Congress to depend on information provided from executive branch bodies such as the Center for AI Standards and Innovation or USAi. Congress needs a body it alone controls and can trust. For now, let’s call it the Congressional AI Research Office (CAIRO).

CAIRO could act as a hybrid between a congressional support agency (such as the OTA) and committee staff. Similar to congressional support agencies, CAIRO should have a deep, standing bench of technological experts. But like congressional committee staff, CAIRO should be explicitly divided into majority and minority staff, with some shared nonpartisan staff for work with less partisan valence. This would enable each party to receive trusted input on the issues they care about. And similar to committee staff, the majority party should be entitled to more funding, but with significant protections for the minority party. This balances the political incentives of the majority party with the ability to retain a stable core of experts across Congresses.

CAIRO staff would analyze proposed deployments for their respective principals. To do this, each side of CAIRO would need to develop extensive in-house expertise on AI systems. This would include not just measuring AI models’ capabilities and propensities but also having the ability to systematically assess all aspects of a proposed AI system, including the development process and other system-level characteristics that may affect the product’s functionality and security.

CAIRO staff might develop and perform their own evaluations but would also need to rely heavily on evaluations from third parties and the executive branch. Accordingly, the staff would need to be able to interrogate the scientific validity and integrity of those external evaluations. And it could also serve as a secure staging environment for members to interact with AI system prototypes or demonstrations. It would also include cleared staff and secure computing environments to assess classified proposed deployments.

These resources are unlikely to wake Congress from its current slumber. But they could represent a necessary first step toward reestablishing a healthier interbranch dynamic.

*          *          *

The future of U.S. state capacity requires ambitious government deployment of AI, and American liberty requires careful regulation of that deployment. Managing this tension will be a central task in the coming decades.

While no single mechanism will suffice, active congressional oversight of the executive branch’s AI use will remain indispensable. Given the speed and scale at which AI can act, prospective approval will be at least as important as retrospective review. Before the executive branch is allowed to use AI for its most significant functions—those that threaten to deprive the people of their life, liberty, and property—the specific consent of the people’s representatives should be obtained.

Annual Report 2025

Executive Summary

The Institute for Law & AI (LawAI) engages in three core activities:


In 2025, our activities grew substantially. Our core team grew from 20 to 28 staff. Applications to our Seasonal Research Fellowship were 9.5x higher than in 2024. We also launched three flagship events, featuring 200+ participants and 50+ experts and moderators.

Meanwhile, our Director, Christoph Winter, served in a legal advisory role to the European AI Office on the General-Purpose AI Code of Practice. He also testified before the European Parliament, offering recommendations on law-following AI and institutional capacity-building at the European AI Office.

Our US Law & Policy team produced 7 new publications, including a report on mobilizing technical talent into government for AI-related national security crises, and discussed implications of the Executive Order on AI preemption with CNN, the Financial Times, Politico, NBC, and The Verge.

Our Legal Frontiers team published a pioneering research article on law-following AI, and has begun to catalyze a field of technical, legal, and policy research to develop the idea more fully, including through our own subsequent research and our Workshop on Law-Following AI.

In 2026, our work will emphasize “radical optionality”— research and policy approaches designed to preserve flexibility amid uncertainty about AI’s trajectory. Our EU Law team will continue to publish new chapters of the Cambridge Commentary on EU General-Purpose AI Law, while our Legal Frontiers team will expand its new Automated Governance workstream. Our fellowship programs are expanding across all tracks with our inaugural Winter Research Fellowship.

Read the full report using the Full text PDFs link in the right sidebar.

Proceedings of the 2025 Workshop on Law-Following AI

Abstract

The inaugural Workshop on Law-Following AI (LFAI), hosted by the Institute for Law & AI at the University of Cambridge from August 6–8, 2025 with support from the Leverhulme Centre for the Future of Intelligence and the UK’s Advanced Research & Innovation Agency, convened more than forty scholars from law, computer science, and related disciplines to advance the emerging research agenda around LFAI: a concept denoting agentic AI systems designed to refuse illegal orders and illegal means, the corresponding policy proposal to mandate such design in certain deployment contexts, and the interdisciplinary field of inquiry that supports both. Rather than recording consensus, these proceedings synthesize key themes from the workshop’s presentations and discussions, including the promises and limits of liability (particularly for governmental AI agents), the state and nuances of automated legal reasoning and evaluation, risks posed by automated compliance and “perfect enforcement,” the appropriate standard of care for AI agents, and the interplay between AI agents and their principals, including fiduciary framings. The report is intended to extend the workshop’s conversations to a broader audience and catalyze further scholarship on LFAI’s design, evaluation, and governance. A full list of report authors can be found at the end of this document. 

Radical Optionality: Governing Transformative AI Under Uncertainty

We are pleased to announce a new essay by LawAI Director Christoph Winter and Senior Research Fellow Charlie Bullock: Radical Optionality: Governing Transformative AI Under Uncertainty.

The prospect of “transformative AI” appears to present policymakers with a dilemma: overregulation could stifle innovation and forfeit the potential benefits of the technology, while a failure to regulate appropriately could have disastrous implications for public safety and national security.

It’s true that security and innovation are sometimes in tension. Some safety measures do impose costs on innovation, and some forms of deregulation do carry genuine risks. But there is also a class of policies that would meaningfully increase safety without imposing significant costs on innovation. We argue that governments should aggressively implement these policies; this is the main thrust of the governance strategy discussed in this essay, which we call “radical optionality.”

At its core, radical optionality is about preserving democratic governments’ ability to make good decisions about how to govern transformative AI systems as circumstances evolve. In the short term, this means avoiding overregulation while rapidly building the institutions, information channels and legal authorities needed to respond competently to a broad range of scenarios.

The argument for focusing on optionality is simple, and—if you accept a few reasonable assumptions—compelling. These assumptions are:

  1. That there is a real possibility of transformative AI (defined as “AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution”) being developed within the next ten years;
  2. That profound uncertainty exists as to what capabilities transformative AI systems will possess, what benefits and risks they will generate, and what the best ways for society to capture the benefits while mitigating the risks will be;
  3. That a transformatively impactful dual-use technology with significant national security implications will inevitably require some degree of government oversight; and
  4. That building the institutional capacity required to effectively govern transformative AI systems will take years, and that society therefore cannot afford to wait until transformative capabilities have actually been developed.

Justifying the first assumption is beyond the scope of this paper. Whether “AGI” or “superintelligence” or “powerful AI” or “transformative AI” will ever arrive, and when, are questions that have been debated extensively elsewhere. But if you believe that transformative AI is possible, we hope to demonstrate that the case for radical efforts to preserve optionality is overwhelmingly strong.

Read the essay at https://radical-optionality.ai/

Announcing the Cambridge Commentary on EU General-Purpose AI Law

We are pleased to announce the launch of the Cambridge Commentary on EU General-Purpose AI Law, the first scholarly commentary focused exclusively on the general-purpose AI model provisions of the EU AI Act. It is a collaboration between the Cambridge Programme on AI Science & Policy at the University of Cambridge and the Institute for Law & AI.

The EU AI Act is the first comprehensive legal framework designed specifically to govern AI. Where existing law — product liability, data protection, sector-specific regulation — reached AI incidentally, the AI Act subjects it to a dedicated legal regime, with all the promise and risk that entails. As a pioneer regulation, it aims not only to promote safe, human-centric and trustworthy AI within the EU, but also to influence global standard-setting, echoing the “Brussels effect” seen in other regulatory domains.

Rigorous legal analysis of the AI Act therefore matters beyond the EU. Robust insights may not only support an effective EU regulatory approach, but will also be relevant to evaluating how best to design effective AI governance regimes in other jurisdictions. In particular, research focused on general-purpose AI (“GPAI”) models, especially those that present systemic risk, may be of particular importance, given the potential for good and harm, technological novelty, and the consequential regulatory uncertainties.

“Many of the most consequential provisions governing frontier or general-purpose AI admit of more than one reasonable interpretation,” said Christoph Winter, general editor of the Cambridge Commentary, Director of the Institute for Law & AI, and Assistant Professor at the University of Cambridge. “Our goal was to be precise about where that uncertainty lies and to lay out the strongest arguments on each side so that readers can form their own views as the law and technology develop.”

Rather than advocating throughout for a single preferred reading, the Cambridge Commentary on each provision aims to map the interpretive landscape: to identify where the legislation is clear, where it is ambiguous, and what the strongest arguments on each side look like. Where contributors favour a particular interpretation, they say so explicitly. The goal is to equip readers — practitioners, regulators, and scholars alike — to reason well about these questions and to adjust their views as the law and technology develop, rather than to hand them conclusions that may not endure over time.

The Cambridge Commentary launches with Chapter V of the AI Act, which forms the core of the GPAI model provisions. Further articles and chapters will be added on a rolling basis.

Access the Cambridge Commentary at cambridge-commentary.ai.

Foreseeing the Unforeseeable: How U.S. Negligence Law Should Address the Foreseeability of Harms Caused by Autonomous AI Agents

Abstract

As AI systems increasingly perform tasks with limited human oversight, courts will soon be required to determine how negligence law should respond when autonomous AI agents cause personal injury. Traditional foreseeability doctrine, as many scholars have observed, may fail to account for the opacity and unpredictability that characterize these systems. The main challenge could arrive when AI developers claim that a harmful outcome was unforeseeable because the specific causal pathway was novel, complex, or obscure. This article argues that such reasoning misallocates responsibility. Building on recent scholarly work, it takes the view that opacity and unpredictability are not inherent features of advanced AI systems. Rather, they are the result of abstraction choices made by AI developers, who often prioritize accuracy and efficiency over interpretability and predictability. When those choices increase the likelihood of opaque or unexpected outcomes, the legal framework should be adjusted to reflect that responsibility. In particular, where the foreseeability standard in negligence law typically makes it difficult for plaintiffs to prevail, it should be relaxed in their favor. The article examines how U.S. courts apply foreseeability across duty, breach, and proximate cause, and identifies the duty stage as the most urgent point for reform. It proposes a three-part doctrinal framework for cases involving personal injury caused by autonomous AI agents. First, courts should preserve existing law where foreseeability is already sympathetic to plaintiffs. Second, they should replace overly fact-intensive duty inquiries with clear, plaintiff-friendly categorical reasoning. Third, they should retain fact-intensive analysis at the breach and proximate cause stages to prevent overextension of liability. This approach maintains foreseeability as a meaningful constraint while calibrating it to the distinctive risks posed by autonomous AI agents.