California's AI School Policy Arrives After a Homework Assignment Backfired
California's new voluntary AI model policy for schools follows an incident where an AI image tool sexualised a child's homework, and critics say the guidance still lacks real guardrails.
A homework assignment goes wrong
In December 2024, fourth-graders at Delevan Drive Elementary School in Los Angeles were set a straightforward assignment: design a book cover for Pippi Longstocking using an AI image generator built into Adobe Express for Education. One student typed a description of the character — "long stockings, a red-headed girl with braids sticking straight out" — and the tool returned sexualised images of women in lingerie and swimwear.
The child's father, Jody Hughes, went public with the incident, according to a CalMatters report. Los Angeles Unified said the images did not meet district standards and that it was working with Adobe to fix the problem. Adobe's vice president of education, Charlie Miller, said the company pushed a fix within 24 hours of being told. The assignment itself was never at fault for asking children to describe a character — the tool's output was.
That story broke in February 2026, in the same window that the California Department of Education released a revised set of AI guidelines. Four months later, on 25 June 2026, the state published something more formal: a Model Policy on Artificial Intelligence in Education, built by a working group convened under Senate Bill 1288 and covering roughly 30 sample districts' worth of legal, ethical and logistical ground.
What the policy actually says
The model policy is not short on scope. It covers academic integrity and disclosure expectations, limits on the reliability of AI detection software, "privacy by design" principles for tools used with student data, legal compliance, educator discretion over whether AI is permitted on a given task, parent and guardian review rights, and AI literacy as a named academic competency.
Crucially, it is voluntary. The California Department of Education states plainly that compliance with any information or guidance in the document is not mandatory, and encourages local educational agencies to adapt it to their own communities. That puts California in contrast with Ohio, which made a written district AI policy a statutory requirement from 1 July 2026. California is choosing template over mandate.
The gap between a state publishing best-practice language and a classroom tool actually behaving safely is exactly where the Delevan Drive incident happened.
Why the criticism has teeth
Education researchers and some parents, including Hughes, have argued the guidance is too vague to have prevented what happened to his daughter. The document encourages students to think critically and creatively with AI but, according to critics cited by CalMatters, offers no specific technical guardrails — no requirement that image generators used with minors be filtered for a defined risk category, no mandated opt-out mechanism for parents uncomfortable with generative tools being used in any subject.
That distinction matters for school leaders anywhere, not just in California. A policy that names good values — privacy, discretion, disclosure — without specifying who tests a tool, how, and against what standard before it reaches a classroom, is a policy that describes the problem rather than closing it. The Adobe incident was fixed in a day once it was public. The question California's critics are raising is why nobody caught it before an assignment went out to nine-year-olds.
The UK angle
England does not have a single state education department the way California does, but the same design choice is playing out here: guidance versus mandate, and who is responsible for vetting a tool before it reaches a lesson plan. Ofsted, the DfE and JCQ have all issued advice this year, but none of it currently requires schools to demonstrate that a third-party AI tool has been safety-tested for the age group using it, rather than simply reviewed for data protection compliance.
The Delevan Drive case is a reminder that data privacy and content safety are different risks. A tool can be GDPR-compliant, log nothing, and still generate the wrong thing when a ten-year-old types an innocent prompt.
What to do
Before any generative AI image, video or voice tool is used with a class, someone at the school should run the exact prompts students are likely to type and see what comes back, not the vendor's demo prompts, the real ones. Written AI policies should specify who tested a tool, when, and for which age group, not just that the school has a policy. Parents should be told which subjects use generative tools and be given a straightforward way to opt a child out of specific tasks, not just the tool in general.
What to watch
Whether California's voluntary approach produces genuinely different classroom practice than Ohio's mandatory one is worth tracking over the next school year, since it is a live natural experiment in whether guidance or law changes behaviour faster. Also watch whether Adobe, Microsoft, Google and other vendors supplying education editions of consumer AI tools start publishing their own age-appropriate content filtering standards, rather than leaving schools to discover the gaps by accident.