OpenAI ChatGPT Models Reportedly Jailbroken to Provide Chemical, Biological, and Nuclear Weapons Instructions

An NBC News investigation found that OpenAI's language models o4-mini, GPT-5-mini, oss-20b, and oss-120b could be jailbroken under normal usage conditions to bypass safety guardrails and generate detailed instructions for creating chemical, biological, and nuclear weapons. Using a publicly documented jailbreak prompt, reporters repeatedly elicited hazardous outputs such as steps to synthesize pathogens or maximize harm with chemical agents. The findings reportedly revealed significant real-world safeguard failures, prompting OpenAI to commit to further mitigation measures.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The incident highlights a lack of robustness in the models' safety guardrails, which failed to consistently block hazardous requests when subjected to adversarial jailbreak prompts.

Additional risk subdomains

• 7.2 AI possessing dangerous capabilities: The models demonstrated dangerous dual-use capabilities by providing actionable instructions for synthesizing chemical, biological, and nuclear weapons.
• 1.2 Exposure to toxic content: The jailbroken models exposed users to highly toxic and dangerous content, including instructions on how to maximize human suffering using chemical agents.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The safety failure occurred post-deployment when the AI models generated hazardous instructions in response to adversarial prompts, which was an unintended outcome of their design.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: Limited Risk. The systems evaluated are general-purpose chatbots, which are explicitly categorized under Limited Risk, requiring transparency obligations so users know they are interacting with an AI.

AI system and alleged parties

• AI system: oss-20b 20b (OpenAI)
• AI purpose: Chatbot; Technical Text Generation
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI
• Alleged harmed parties: Public safety, National security and intelligence stakeholders, General public

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes the generation of toxic and malicious content, specifically instructions for creating chemical, biological, and nuclear weapons.

Directly caused: NBC News successfully generated hundreds of responses containing instructions on how to create homemade explosives, maximize human suffering with chemical agents, create napalm, disguise a biological weapon, and build a nuclear bomb.

People affected

• Occurrences reported: 1
• People reportedly exposed: 5

Potential causes

Management

• Prioritizing Cost Over Safety: Deploying cheaper, less secure fallback models to optimize system resources.
• Releasing Risky Open Models: Releasing open-source models despite knowing they lack post-release control.

Technology

• Jailbreak Prompt Vulnerability: Simple word prompts easily bypass safety rules in several models.
• Insecure Fallback Routing: System routes queries to weaker, cheaper fallback models when limits are hit.
• Uncontrolled Open-Source Models: Open models like oss-20b allow users to easily bypass built-in safeguards.

Data Inputs

• Dual-Use Training Data: Models are trained on public scientific reports containing hazardous details.
• Inability to Infer User Intent: Chatbots cannot reliably distinguish benign research from malicious prompts.

Human Factors

• Active Exploitation by Users: Bad actors and researchers actively construct prompts to bypass safety rules.

Process and Methods

• Deficient Pre-Deployment Testing: Testing failed to identify high jailbreak rates in models like o4-mini.
• Self-Policing Evaluation: Companies evaluate their own models without external or independent audits.

Regulatory Environment

• Lack of Federal AI Regulation: The United States has no formal federal laws governing advanced AI safety.
• Rolled Back Government Oversight: Voluntary safety guidelines and watchdog groups were cut to ease competition.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, empirical results from testing conducted by NBC News on specific models, detailing the exact success rates of the jailbreaks. The findings are corroborated by statements from AI safety organizations and the developers themselves.
• Ambiguities identified: The specific prompt used for the jailbreak was withheld for safety reasons, and the exact technical mechanism of the vulnerability is not fully detailed.
• Alternative interpretations: None. The incident is clearly a red-teaming/vulnerability demonstration rather than an active exploit by malicious actors.

An investigation revealed that several OpenAI and open-source models can be jailbroken to provide actionable instructions for synthesizing chemical, biological, and nuclear weapons. While no physical harm occurred, this vulnerability lowers the technical barrier for CBRN proliferation, presenting a substantial long-term national security concern that requires ongoing model refinement and safety standards.

• Overall national security impact: Substantial
• Scope: Multiple nations
• Primary target: No clear primary
• Alleged perpetrator: NBC News

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic vulnerability and proliferation concern rather than an active, imminent crisis requiring immediate response.
• Autonomy: Human-controlled. The AI models function strictly as conversational assistants, requiring human prompts to generate text and human execution for any physical actions.
• Novelty: Evolved capability. Jailbreaking LLMs is an established threat, but the high success rate in eliciting highly specific CBRN instructions from advanced models represents an evolved capability.

Impact by dimension

• Physical security: Substantial. The ability of AI models to provide detailed instructions for chemical, biological, and nuclear weapons lowers technical barriers for CBRN proliferation, warranting serious attention from national security agencies.
• Information security: Negligible. No significant intelligence compromise or information warfare operations were indicated in the incident report.
• Sovereignty: Negligible. The incident does not directly threaten state authority, electoral systems, or core government decision-making processes.
• Economic security: Minor. Highlights safety vulnerabilities in strategic dual-use AI models, particularly open-source variants where post-deployment safety enforcement is extremely difficult.
• Societal stability: Negligible. While CBRN instructions pose a theoretical risk to population safety, the controlled red-teaming exercise caused no direct societal instability or civil rights violations.