OpenAI Launches Free CSAM Image Classifier and Adds ChatGPT Warnings With 24-Hour Bans

OpenAI announced on November 16 a free image classifier that detects child sexual abuse material, including AI-generated images, alongside new ChatGPT warning screens and 24-hour account suspensions, plus developer guidance and AI tools for law enforcement investigators.

Nov 17, 2026 - 23:34
 0  3
OpenAI Launches Free CSAM Image Classifier and Adds ChatGPT Warnings With 24-Hour Bans
A smartphone displaying the ChatGPT logo in front of a large blurred ChatGPT logo on a blue background.

OpenAI Launches Free CSAM Image Classifier and Adds ChatGPT Warnings With 24-Hour Bans

OpenAI has unveiled a new set of child-safety tools aimed at child sexual abuse material, including a free image classifier that can spot abusive images it has never seen before, new warning screens inside ChatGPT, and guidance for developers and law enforcement. The company announced the package on November 16, as pressure grows on AI companies over how their products can be misused to create and spread abusive imagery.

The centrepiece is a classifier built to detect child sexual abuse material, known as CSAM, in images — including images generated by AI. OpenAI says it is the first broadly available CSAM classifier with a dedicated signal for spotting whether material may have been AI-generated, a distinction that matters because synthetic imagery is now a fast-growing part of the problem.

A classifier that spots abuse it has never seen before

Most existing detection relies on matching images against databases of material that has already been identified, such as hash-matching systems. That approach cannot catch something new. OpenAI's classifier works differently: it looks for visual signs of abuse in images it has not encountered before and assigns each image a score, helping human reviewers decide which images to examine first.

OpenAI says it developed and evaluated the tool with the National Center for Missing and Exploited Children (NCMEC), the US nonprofit that runs the CyberTipline, the central reporting system for online child exploitation. The company is offering the classifier free through its API to vetted organisations, and is working with the Tech Coalition and ROOST, two industry safety groups, to widen access. OpenAI has also joined the Tech Coalition's Lantern programme for sharing child exploitation signals across platforms.

In early testing with partners, OpenAI says the classifier flagged 48 percent fewer images for human review while finding roughly the same number of abusive images, and that the images it did flag were about twice as likely to be genuinely abusive. Those figures come from OpenAI's own early partner testing, not an independent audit, so they should be read as the company's claim rather than a proven result.

Partner results reported in coverage of the announcement give a sense of how the tool performed in practice. Reviewers at fal confirmed 72 percent of the classifier's high-confidence flags as previously unidentified abusive material, according to reports, while invideo AI said one in four flagged cases ended in a ban, and SafetyKit said it detected 90 percent of drawings and animations that sexualise children.

Warnings and 24-hour suspensions inside ChatGPT

Inside ChatGPT, OpenAI is adding warning screens for users whose prompts seek content that sexualises children. Instead of simply blocking the request, the app will show an alert, point the user toward specialised support services designed to stop harmful behaviour, and suspend the account for 24 hours as a warning. ChatGPT already blocks CSAM content and related requests outright, and OpenAI says repeated offences can lead to the account being removed permanently.

The company says it shaped the warning messages and guidelines with outside child-protection experts, including Protect Children, a Finnish child welfare organisation, the Lucy Faithfull Foundation, a UK charity that works to prevent child sexual abuse, and Joel Scanlan, a researcher at the University of Tasmania who independently evaluated a trial of deterrent messages.

New guidance for developers

OpenAI is also publishing guidance for developers building on its tools on how to prevent, detect and report CSAM on their platforms. The guidance was developed with input from the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation, and OpenAI says early testing showed it helped developers adopt child-safety tooling and register to report CSAM to NCMEC. The company has also worked with Anthropic and Google to commission guidance on responding to disclosures of child sexual exploitation and abuse in AI systems.

Tools and training for investigators

For law enforcement, OpenAI is providing AI-powered features including dedicated plug-ins, investigative tools and complex data analysis, with access managed by programme administrators, along with training for investigators and a field guide on how to use AI in child-safety work. NCMEC senior vice president John Shehan said the technology could do more than identify material — it could help find the children who may be depicted and give investigators information to protect them.

Why now: the scale of AI-generated abuse

The announcement lands as the volume of AI-related abuse reports is climbing sharply. NCMEC's CyberTipline received more than 400,000 reports involving generative AI in 2025, up from 67,000 the year before, according to figures cited in coverage of the announcement. North Carolina attorney general Jeff Jackson described the increase as explosive and warned that without new tools the reporting system could become unmanageable.

OpenAI has faced growing scrutiny over child safety. The company has had several lawsuits filed against it blaming its technology for harming users, especially children, and safety-focused nonprofit Common Sense Media labelled OpenAI's technology an unacceptable risk for teens. Other AI chatbots have also been misused — Grok has been used to create nonconsensual sexualised imagery, including of children. Over the past 12 months, OpenAI says it has released a safety blueprint, updated its content guidelines, retuned its models to prevent them from generating harmful content, and released a special version of ChatGPT for teens.

The new classifier will not solve the problem on its own. It is available only to vetted organisations, its headline figures have not been independently verified, and a score from a classifier does not by itself confirm that an image is abusive or establish what the law requires in any jurisdiction. But by giving the tool away free and pairing it with warnings inside ChatGPT and guidance for developers, OpenAI is betting that earlier detection and clearer deterrents can slow abuse before it spreads. OpenAI has also been expanding what ChatGPT itself can do, adding a wallet that lets AI agents shop and check out on a user's behalf.

Sources

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Nahid Joy I’m Nahid Hasan Joy, a technology writer, web developer, and digital enthusiast with a strong interest in the ever-changing world of technology.