Chatbot Tessa gives unauthorized diet advice to users seeking help for eating disorders

The National Eating Disorders Association (NEDA) has shut down its chatbot named Tessa after it gave weight-loss advice to users seeking help for eating disorders. The incident has raised concerns about the risks of using chatbots and AI assistants in healthcare settings, particularly in addressing sensitive issues like eating disorders. NEDA is investigating the matter, emphasizing the need for caution and accuracy when utilizing technology to provide mental health support.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The chatbot failed to perform reliably and safely under user interaction, generating harmful and medically inappropriate advice due to an unapproved generative AI upgrade.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: NEDA attempted to transition a critical helpline service to an automated chatbot, representing unsafe use of AI in a sensitive mental health context.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The immediate cause of the risk was the AI chatbot generating unscripted, harmful weight-loss advice to vulnerable users post-deployment, which was an unintentional outcome of its programming.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a chatbot designed to interact with users, which falls under the Limited Risk category requiring transparency obligations so users know they are interacting with an AI.

AI system and alleged parties

• AI system: Tessa
• AI purpose: Health AI Assistant; Chatbot
• Behaviour type: Assistant
• Alleged developer: Cass
• Alleged deployer: National Eating Disorders Association, Cass
• Alleged harmed parties: People with eating disorders

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes the chatbot generating and spreading harmful weight-loss and dieting advice to eating disorder patients.

Directly caused: The chatbot directly generated unscripted advice recommending calorie deficits of 500 to 1000 calories and the use of calipers to measure body fat.

Inferred additional harm: The chatbot likely generated similar harmful content in other unmonitored interactions during the period the generative features were active.

Psychological

Reported: The report explicitly describes psychological distress and triggering of eating disorder behaviors among users.

Directly caused: Vulnerable users and eating disorder specialists reported receiving weight-loss and calorie-restriction advice that directly triggered disordered eating thoughts.

Inferred additional harm: Given the 25,000 messages exchanged, it is highly likely that dozens of other unquantified users experienced psychological distress or setbacks in their recovery.

People affected

• Occurrences reported: 1
• People reportedly harmed: 25
• People reportedly exposed: 5200

Potential causes

Management

• Uncoordinated Vendor Management: NEDA did not maintain visibility into Cass's technical updates.
• Premature Helpline Closure: Management closed the human helpline before validating the chatbot.
• Growth Prioritization Over Safety: NEDA sought to scale prevention tools rapidly without proper risk controls.

Technology

• Unsanctioned Generative AI Upgrade: Cass added generative AI features without NEDA's knowledge or authorization.
• Off-Script Response Generation: The chatbot went off-script and generated unapproved weight-loss advice.
• Inability to Detect Eating Disorder: The bot failed to recognize user disclosures of active eating disorders.

Data Inputs

• Mismatched Training Datasets: Generative models used standard diet advice unsuitable for eating disorders.
• Lack of Input Validation for Crisis: The system did not properly filter or flag crisis-related user inputs.
• Nefarious Inputs by Bad Actors: Bad actors actively tried to trick the bot during a massive traffic surge.

Human Factors

• User Expectation of Empathy: Vulnerable users expected human-like empathy from an automated chatbot.
• Lack of Professional Supervision: No human professionals monitored the chatbot interactions in real-time.
• Overreliance on Automated Advice: Users trusted the bot's advice despite its inappropriate recommendations.

Process and Methods

• Inadequate Integration Testing: The upgraded generative features were not thoroughly audited before deployment.
• Lack of Collaborative Review: NEDA and developers failed to co-verify the bot's updated content.
• Deficient Safety Guardrails: The system's guardrails failed to block restrictive dieting tips.

Regulatory Environment

• Mandated Reporting Liabilities: Complex state and federal reporting rules drove NEDA to replace humans.
• Absence of AI Healthcare Standards: Lack of clear regulations for deploying AI in mental health settings.
• Lack of AI Certification Rules: No regulatory body certified the chatbot safe for mental health use.

Information quality

• Classification confidence: High
• Reason for confidence: Multiple high-quality investigative reports from major outlets (WSJ, NYT, NPR) provide consistent accounts of the chatbot's behavior, the organizational context of NEDA, and statements from both NEDA and the developer Cass. The technical cause (unauthorized generative AI upgrade) is clearly identified.
• Ambiguities identified: There is slight conflicting reporting on whether NEDA intended Tessa to fully replace the helpline or run as a separate service, though internal audio suggests a planned transition.

The NEDA wellness chatbot Tessa was taken offline after an unauthorized generative AI upgrade led it to provide harmful weight-loss advice to users. While representing a significant failure in AI safety and vendor oversight within a sensitive health context, the incident has negligible national security implications.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The chatbot has been taken offline, eliminating any immediate threat, though it highlights long-term strategic concerns regarding AI safety in healthcare.
• Autonomy: Full autonomy. The chatbot generated and delivered unscripted responses directly to users without real-time human review or intervention.
• Novelty: Established threat. Generative AI systems hallucinating or providing inappropriate advice is a well-documented and established threat.

Impact by dimension

• Physical security: Negligible. The incident involved a health wellness chatbot and had no impact on physical systems, critical infrastructure, or kinetic capabilities.
• Information security: Negligible. No classified information, intelligence collection methods, or state-sponsored information operations were involved.
• Sovereignty: Negligible. The incident did not impact state authority, electoral systems, or government decision-making processes.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or critical supply chain disruptions occurred.
• Societal stability: Negligible. While the chatbot provided harmful advice to a small group of vulnerable individuals, it did not pose a large-scale threat to societal stability, civil liberties, or population safety.