Don’t Let AI Developers Hire Their Own Referees
Letting AI developers pick their own safety auditors creates a conflict of interest. Requiring liability insurance instead would put insurers’ own capital behind risk assessments.
This piece was originally published by AI Frontiers.
Introduction
A growing chorus of scholars and policymakers favors letting private organizations—rather than a government regulator—govern frontier AI. In the leading family of proposals, the state sets the safety outcomes it wants and licenses independent verification organizations (IVOs) that compete to certify developers against those outcomes. Gillian Hadfield has developed the idea as “regulatory markets,” in which AI developers must pay for oversight from private regulators that governments license and hold accountable for safety standards. Dean Ball, who likens the arrangement to bank supervision, has argued for a version he calls “private governance,” which a nonprofit named Fathom has converted into model legislation. The rationale is that legislators and agencies are poorly positioned to write good safety rules for frontier AI: they understand these systems less well than the labs building them, and rules fixed in advance cannot keep pace as the technology changes. Private verifiers, meanwhile, are closer to the technology than any agency and are disciplined by competition, so they can set better technical standards and keep them current.
The model is no longer hypothetical. The bipartisan FRONTIER Act, introduced in the House in July as the successor to the Great American AI Act discussion draft, would require the largest frontier developers to retain licensed IVOs that audit their risk-management efforts and report to federal overseers. California’s SB 813, backed by Fathom, would have let developers earn a shield from tort liability if they met standards set by a private organization accredited by the state attorney general. It failed this session, but similar proposals are likely to return. Virginia has directed a state commission to study the IVO model for AI regulation. And Connecticut has gone furthest: its omnibus AI law enacted this spring creates a multiyear pilot under which the state consumer-protection department may approve up to five IVOs, whose certification would help companies in court without entirely shielding them from liability.
Unfortunately, as currently structured, IVO-based governance has a key design flaw. Under the regulatory frameworks mentioned above, AI developers would typically select and pay the organizations that certify them, giving IVOs a financial incentive that might clash with high safety standards. This essay will explain how such incentives can interfere with good governance and outline an alternative model of regulation that builds in the right incentives through mandatory insurance.