OpenAI tightens ChatGPT weapons safeguards for teens amid violence-prevention push

OpenAI is adding stricter limits to the weapons-related information ChatGPT provides to teenagers, including when a question does not appear to have violent intent. The company said the changes are part of a broader effort to reduce the risk that its chatbot could contribute to real-world violence.

Broader limits on weapons and attack details

OpenAI said ChatGPT for Teens will restrict practical information about weapons and security systems that could be misused to facilitate violence. The company said teenagers will still be able to use the service for violence prevention, de-escalation and support, including help connecting with a trusted adult.

The company is also tightening responses for users of any age about past mass attacks. ChatGPT will provide less detail about weapons and equipment used, how they were obtained and other reconstruction details that could increase an attacker’s notoriety, according to OpenAI. Users can still ask about the events’ impact, warning signs and prevention.

Chats may be interrupted when violence becomes a fixation

For all users, OpenAI said ChatGPT will be able to end a conversation automatically when recent exchanges suggest that someone is fixating on violence or seriously harming another person. If multiple chats are ended for that reason, the company may temporarily pause the account. OpenAI said these safeguards will begin rolling out over the next few weeks and will continue to be refined.

OpenAI also said it has strengthened its age-prediction system, which assesses signals such as behavior patterns and can move an account into the under-18 experience. Parents with linked accounts already receive limited safety notifications, and OpenAI said it is working toward notifying parents when a teen’s account is paused after repeatedly discussing violence.

Scrutiny over protections for young users

The measures follow increasing scrutiny of AI companies’ ability to detect and respond to threats involving young users. Reuters reported that OpenAI said teens spend less than 15 minutes a day on ChatGPT on average in the company’s first report on teen use. A Common Sense Media report cited by Reuters said the teen version often alerted parents late or not at all in conversations involving suicide, self-harm or eating disorders; that finding is a report’s assessment, not an established fact about every account.

Sources

• Reuters: OpenAI to limit weapons information for teens on ChatGPT in violence-prevention push
• OpenAI: Community safety report and new protections for teens
• OpenAI: Introducing ChatGPT for Teens