California's SB 1047 forces a choice: regulate AI models or their harms

California’s SB 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, is about to become law — and it has forced a question no one has answered cleanly: should lawmakers regulate the model itself or the harms it enables? A September 2024 AI Pulse report from Trend Micro captures the unsettled state of play, as legislators in the EU, UK, and United States signed the first legally binding AI treaty while experts at the European Center for Not-for-Profit Law called it toothless.

The report is a useful snapshot of a moment when frameworks are proliferating faster than consensus. SB 1047 applies to models costing $100 million or more to build, trained on at least 10^26 floating-point operations. Everyone involved acknowledges those thresholds are imprecise measures of threat potential. The bill drew support from AI luminaries Geoffrey Hinton and Yoshua Bengio, while Andrew Ng told Forbes it makes a “fundamental mistake” by regulating AI as a technology instead of focusing on specific applications.

That distinction — model versus application — is the central tension in every jurisdiction now writing rules. The Trend Micro report notes that financial services and healthcare are heavily regulated sectors and also leaders in AI adoption, citing Canadian research on AI reducing unexpected hospital deaths. The implication is that regulation does not automatically kill innovation. But the comparison is sloppy. Healthcare regulation targets clinical outcomes and data privacy, not the computational architecture of a neural network. SB 1047 targets the latter.

OpenAI’s release of a system card for its o1 model in September illustrates the measurement problem. The company ranked o1 low-risk on autonomy and cybersecurity, medium-risk on persuasion and chemical, biological, radiological, and nuclear (CBRN) dangers. Anything medium or lower is considered deployable. Bengio told Newsweek that o1’s medium-risk CBRN score “reinforces the importance and urgency to adopt legislation like SB 1047 in order to protect the public.” The Trend Micro report also flags that o1’s deceptive capabilities have increased, raising concerns of “Rogue AI.”

Two things are true at once. The o1 scorecard is more transparency than most labs offer. And a medium-risk rating on CBRN, combined with a company policy that says medium is deployable, is not a reassuring basis for letting a model loose. The scorecard is self-reported. There is no independent verification. That is the gap SB 1047 tries to fill, however clumsily.

A more promising development is the late-August agreement between the National Institute of Standards and Technology (NIST), OpenAI, and Anthropic to collaborate on AI safety research, testing, and evaluation. That is a concrete mechanism: government standards body plus private labs, working on shared benchmarks. It is not regulation in the traditional sense, but it may produce the measurement infrastructure that regulation requires.

The military domain is further along in some ways. The second REAIM Summit in South Korea in September produced a Blueprint for Action with 20 principles for military AI use, including that “humans remain responsible and accountable for [AI] use and [the] effects of AI applications in the military domain, and responsibility and accountability can never be transferred to machines.” Not all countries signed. The Times of India ran a provocative headline: “China refuses to sign agreement to ban AI from controlling nuclear weapons.” The truth is more nuanced, but the gap between signatories and holdouts maps directly onto the broader regulatory problem: without shared definitions of risk, you cannot get shared rules.

The Trend Micro report also covers the Coalition for Secure AI (CoSAI), spun up by OASIS Open this past summer, and the OWASP Top 10 Checklist for large language model risks. These industry-led efforts are faster and more technically grounded than legislation. They also lack enforcement power. A company can ignore OWASP’s checklist with no legal consequence.

The data drought is the structural pressure behind much of this. The report cites a paper from dataprovenance.org showing a “rapid crescendo of data restrictions from web sources” in 2023-2024 that will impact commercial and academic AI. Models training on their own output risk what Nature calls “model collapse.” Fresh human data is essential. Clearview AI was hit with a $33 million fine in September for compiling an illegal database of 30 billion images. X’s Grok AI faced European regulatory scrutiny for automatically opting users into data training. The scarcity of clean training data is pushing companies toward behaviors that invite regulation.

The takeaway for AI builders is straightforward. The regulatory window is closing. SB 1047 is imperfect, but it is law. The EU AI Act is coming. The NIST testing agreements are operational. The question is no longer whether models will be regulated, but whether the regulation will be shaped by people who understand the technology or by people who read headlines about rogue AI. Industry participation in standards bodies like NIST and OASIS is not optional anymore. It is the only way to make the measurement problem solvable before the thresholds get set by legislation written without technical input.