Alleged Harmful Health Outcomes Following Reported Use of Purported ChatGPT-Generated Medical Advice in Hyderabad

Reports from Hyderabad describe two alleged patient harms after individuals acted on purportedly ChatGPT-generated medical advice instead of clinician guidance. A kidney-transplant recipient reportedly discontinued prescribed post-transplant medications based on a chatbot suggestion and experienced graft failure. In a separate case, a man with diabetes allegedly developed severe hyponatremia after following a chatbot-advised zero-salt diet.

Risk classification

• Primary risk domain: 5 Human-Computer Interaction
• Primary risk subdomain: 5.1 Overreliance and unsafe use

The primary harm arose because patients placed inappropriate trust and reliance on a general-purpose chatbot for critical medical decisions, treating it as a substitute for professional clinical care.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI system failed to perform safely by hallucinating toxic dietary advice (sodium bromide) and incorrect medical guidance.
• 3.1 False or misleading information: The chatbot generated false medical claims that directly misled patients into taking harmful actions.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The physical harm resulted from ChatGPT generating incorrect and dangerous medical and dietary advice post-deployment, which was an unintentional outcome of its text generation capabilities.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system involved is ChatGPT, which is a general-purpose chatbot. Under the EU AI Act, chatbots are classified as limited risk and are subject to transparency obligations to ensure users know they are interacting with an AI.

AI system and alleged parties

• AI system: ChatGPT (OpenAI)
• AI purpose: Chatbot; Question Answering
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI
• Alleged harmed parties: Unnamed patient with diabetes in Hyderabad, Unnamed kidney transplant patient in Hyderabad, OpenAI users, General public of India, General public of Hyderabad, General public, ChatGPT users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Minor
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Physical

Reported: The report explicitly describes severe physical harm to multiple individuals caused indirectly by acting on the AI's advice.

Indirectly caused: A 30-year-old woman lost her transplanted kidney and returned to dialysis; a 62-year-old diabetic man suffered severe electrolyte imbalance and weight loss; a 60-year-old man was hospitalized after consuming toxic sodium bromide.

Inferred additional harm: It is highly likely that other patients globally have suffered undocumented physical health complications due to relying on incorrect AI-generated medical advice.

Epistemic

Reported: The report explicitly describes epistemic harm where the AI generated false and misleading medical and dietary advice.

Directly caused: ChatGPT generated incorrect medical assertions, such as telling a transplant patient to stop antibiotics and recommending a toxic substance as a salt substitute.

Indirectly caused: Patients believed the fabricated and incorrect information, leading them to make dangerous health decisions.

Inferred additional harm: Widespread use of LLMs for medical queries likely exposes a large number of users to hallucinated or inaccurate health information, eroding trust in verified medical facts.

People affected

• Occurrences reported: 3
• People reportedly harmed: 3
• People reportedly exposed: 3

Potential causes

Management

• Lack of Staff AI Training: Clinics lack structured training on responsible AI and verification safety.
• No Team Policy on AI Use: Absence of clear staff policies for verifying clinician-facing AI outputs.

Technology

• Lack of Clinical Judgment: AI lacks clinical judgment and context that a trained doctor brings.
• Hallucination of Medical Facts: Chatbots hallucinate facts and cite outdated guidance with authority.
• Inability to Detect Interactions: AI misses contraindications, drug interactions, and transplant protocols.

Data Inputs

• Interpretation of Isolated Inputs: Chatbots interpret isolated inputs without looking at trend lines or charts.
• Lack of Patient Longitudinal Data: AI lacks access to patient charts, medication history, and comorbidities.

Human Factors

• Over-reliance on AI Advice: Educated and elderly patients rely on AI-generated advice over doctors.
• Self-Discontinuation of Meds: Patients stop critical treatments based on chatbot outputs without checking.
• Belief in AI Superiority: People start to believe that ChatGPT is superior to medical professionals.

Process and Methods

• Lack of Clinic AI Guidance: Discharge summaries lack warnings against altering meds based on AI advice.
• Absence of AI Triage Path: Clinics lack structured triage paths to review AI advice brought by patients.
• No Medication Teach-Back Rules: Lack of teach-back processes to confirm patients follow prescribed regimens.

Regulatory Environment

• Lack of Chatbot Accountability: AI tools have no duty of care, informed consent, or regulatory follow-up.
• Lack of Standardized AI Governance: Absence of adherence to global ethics and governance standards for health AI.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, consistent, and detailed accounts of three specific medical incidents resulting from ChatGPT advice, including clinical details from senior medical professionals.
• Ambiguities identified: The exact dates of the incidents and the specific prompts used by the patients are not detailed.
• Alternative interpretations: The incidents could be viewed purely as human error due to user negligence, but the AI's generation of highly specific, dangerous advice (like recommending sodium bromide) points to a system safety failure.

Multiple individuals in India and the US experienced severe health complications after relying on medical advice from ChatGPT. While raising serious concerns regarding AI safety, consumer protection, and public health, the incident has negligible direct national security implications.

• Overall national security impact: Negligible
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: India, United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This represents an ongoing consumer safety and public health concern rather than an active national security crisis.
• Autonomy: Human-controlled. The AI acted as a text-generation assistant, and human users made the final decisions to implement the advice.
• Novelty: Established threat. Large language model hallucinations and user overreliance on AI for specialized advice are well-documented, established issues.

Impact by dimension

• Physical security: Negligible. The incident involves isolated cases of individuals suffering health complications due to incorrect medical advice, presenting no threat to critical infrastructure or national physical security.
• Information security: Negligible. No information warfare, intelligence compromise, or state-sponsored disinformation operations are associated with this incident.
• Sovereignty: Negligible. There is no impact on state authority, government decision-making, or core constitutional processes.
• Economic security: Negligible. The incident does not threaten strategic industries, financial systems, or national technological competitive advantage.
• Societal stability: Negligible. While tragic for the individuals involved, these isolated medical incidents do not threaten large-scale social cohesion, civil liberties, or societal stability.