
OpenAI Releases CSAM Detection Tools and Child-Safety Support

OpenAI has released a package of new child-safety measures: a CSAM image classifier available at no cost to vetted organizations through the OpenAI API, new ChatGPT warnings that interrupt attempts to find abusive material, practical guidance for developers, and tools and training for child-safety law enforcement investigators. These build on the company’s Child Safety Blueprint and its existing work to detect, block, and report abuse.
The new classifier is designed to detect potential child sexual abuse material (CSAM), including AI-generated images, by looking for visual signs of abuse and returning scores that help reviewers prioritize which images may need attention. OpenAI says it is the first broadly available CSAM classifier with a specific signal indicating whether CSAM may be AI generated. It was trained, evaluated, and designed with assistance from the National Center for Missing and Exploited Children (NCMEC), which operates the CyberTipline. The classifier is being shared through the Tech Coalition and ROOST, and OpenAI also supports Thorn‘s Safer CSAM detection system and participates in the Tech Coalition‘s Lantern program for cross-platform signal sharing.
Early tests suggest the classifier can reduce review work while finding similar amounts of abusive material. Compared with existing protections on user-uploaded images, OpenAI estimated the new classifier would flag 48% fewer images while finding roughly the same number of abusive images, and images it flagged were about twice as likely to contain abuse. Early access partners also reported results: in fal’s testing on human-reviewed production images, reviewers confirmed 72% of the classifier’s high-confidence flags as novel CSAM; invideo AI found that one in four flagged cases led to a ban; and SafetyKit’s evaluations involving drawings or animations that sexualize children found the classifier detected 90% of cases.
OpenAI also introduced product warnings in ChatGPT that interrupt attempts to find abusive material, explain the consequences, and point people toward help. The warning states that sexualizing children causes harm, including when the requested material is AI-generated. After a flagged attempt, the conversation is removed and access to ChatGPT is paused for 24 hours, while specialist support remains available to help people stop harmful behavior; repeated violations may lead to longer restrictions, account closure, or reports to authorities. This approach was developed with input from organizations including Protect Children and the Lucy Faithfull Foundation, and from researchers who independently evaluated a trial of deterrent messages.
For developers, OpenAI published CSAM guidance covering practical steps to prevent abuse, detect harmful content, and act on it. Developed with input from the Tech Coalition, Thorn, NCMEC, and the Internet Watch Foundation, the guidance provides rules, user reporting options, detection tools, review processes, and advice on using safety identifiers and supporting content reviewers. It is meant to be used alongside tools such as the new classifier, the Moderation API, ROOST‘s Coop review console, and Thorn‘s Safer. OpenAI also worked with Anthropic and Google to jointly commission guidance on responding to disclosures of child sexual exploitation and abuse in AI systems, recently published by the Tech Coalition. According to OpenAI, this guidance has already led to more developers accessing child safety tooling and registering to report CSAM to NCMEC.
Finally, OpenAI is supporting child-safety law enforcement units with new tools and trainings on using AI in investigations. The initial tool helps investigators review lengthy chat logs and legal process returns, identify potential connections, and create timelines with findings linked to source material. The training is planned as an overview and demonstration with a practical field guide, and access will depend on agency authorization and program requirements. Future iterations of the training will be proposed at regional, national, and international conferences.

