ChatGPT Reportedly Suggests Sodium Bromide as Chloride Substitute, Leading to Bromism and Hospitalization

A published medical case report describes a 60-year-old man hospitalized for three weeks with severe bromide toxicity (bromism) after replacing dietary sodium chloride with sodium bromide purchased online. The patient reported making this substitution following consultation with ChatGPT, which allegedly suggested bromide as a chloride substitute without safety warnings. The harm included psychosis, electrolyte imbalances, dermatologic changes, and micronutrient deficiencies.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI system generated misleading and decontextualized information suggesting that chloride could be replaced with bromide, leading the user to form a false belief and ingest a toxic substance.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The user placed inappropriate trust and reliance on a general-purpose chatbot to make critical dietary and health decisions.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by ChatGPT generating a decontextualized and unsafe recommendation post-deployment, which was an unintentional failure of the system's safety filters.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: ChatGPT is a general-purpose conversational chatbot. Under the EU AI Act, chatbots are classified as limited risk and are subject to specific transparency obligations to ensure users know they are interacting with an AI.

AI system and alleged parties

• AI system: ChatGPT (OpenAI)
• AI purpose: Chatbot; Question Answering
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI
• Alleged harmed parties: Unnamed 60-year-old male patient with bromism, ChatGPT users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Minor
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Physical

Reported: The report explicitly describes physical harm to the patient caused indirectly by the incident.

Indirectly caused: A 60-year-old man suffered severe bromide poisoning (bromism) requiring a three-week hospitalization, intravenous fluids, and antipsychotic medication after ingesting sodium bromide based on ChatGPT's advice.

Psychological

Reported: The report explicitly describes severe psychological and mental-health harm caused indirectly by the incident.

Indirectly caused: The patient experienced severe neuropsychiatric symptoms, including paranoia, visual and auditory hallucinations, and a full-blown psychotic episode, which resulted in an involuntary psychiatric hold.

Epistemic

Reported: The report explicitly describes epistemic harm caused directly and indirectly by the incident.

Directly caused: ChatGPT generated scientifically decontextualized information suggesting that chloride could be replaced with bromide, which is factually misleading in a dietary context.

Indirectly caused: The dissemination of decontextualized scientific information via AI can fuel the spread of medical misinformation and lead users to form false beliefs about health and nutrition.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Reliance on Terms of Service: Management relied on disclaimers rather than technical safety blocks.
• Delayed Safety Upgrades: Proactive health-related safety features were not deployed in GPT-3.5/4.

Technology

• Decontextualized AI Output: AI suggested bromide as a chloride swap without verifying the dietary context.
• Lack of Proactive Warnings: The model failed to output safety warnings regarding toxic bromide ingestion.
• No Clinical Probing Ability: The AI did not ask why the user wanted to replace chloride before answering.

Data Inputs

• Inaccurate Scientific Data: AI training data lacked structured medical constraints on chemical swaps.

Human Factors

• User Overreliance on AI: The patient relied on ChatGPT as a sole source of truth for health decisions.
• Self-Experimentation Bias: Patients nutrition background led to a risky personal dietary experiment.
• Lack of Medical Consultation: The patient did not consult a qualified doctor before altering his diet.

Process and Methods

• Inadequate Safety Guardrails: The AI system lacked active filters to block hazardous medical advice.
• No Context Verification Process: The system lacked methods to detect and clarify user intent in queries.

Regulatory Environment

• Unregulated Online Chemical Sales: Sodium bromide was easily purchased online without medical oversight.
• Lack of Medical AI Regulation: Absence of strict regulations governing AI-generated health advice.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly detailed, including a peer-reviewed medical case report from the Annals of Internal Medicine. The medical symptoms, diagnosis of bromism, and the patient's use of ChatGPT are clearly documented, leaving little ambiguity about the sequence of events.
• Ambiguities identified: The exact chat logs between the patient and ChatGPT were unavailable to the doctors, so the precise phrasing of the prompt and response is inferred based on the patient's recollection and the doctors' replication attempts.
• Alternative interpretations: The incident could be interpreted primarily as a human error of overreliance on a non-medical tool, rather than an AI system failure, though the AI's generation of decontextualized advice is a key contributing factor.

An isolated consumer safety incident where an individual suffered poisoning after acting on decontextualized dietary advice from ChatGPT. While highlighting significant consumer safety and LLM alignment risks, the incident has negligible national security implications.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The individual incident is resolved. Any broader national concern is a long-term, ongoing regulatory and safety discussion regarding general-purpose LLM alignment.
• Autonomy: Human-controlled. The AI acted purely as an information assistant. The human user retained complete decision-making authority and voluntarily executed the harmful action.
• Novelty: Established threat. LLM hallucinations, decontextualized responses, and potential for generating unsafe medical advice are well-known and previously documented issues.

Impact by dimension

• Physical security: Negligible. No physical critical infrastructure, defense systems, or mass casualty vectors were targeted or affected. This was an isolated individual medical poisoning.
• Information security: Negligible. No foreign intelligence operations, systematic disinformation campaigns, or classified model thefts occurred. The incident involves a user-initiated medical query.
• Sovereignty: Negligible. Core government operations, electoral systems, and state sovereignty were entirely unaffected by this individual consumer safety incident.
• Economic security: Negligible. No strategic technology theft, financial system disruption, or macroeconomic impact occurred. Financial impact was limited to the individual's medical bills.
• Societal stability: Negligible. The incident is an isolated case of consumer overreliance and did not involve mass surveillance, systemic discrimination, or threats to population-scale social cohesion.