AI Meal Planner Suggests Hazardous Chlorine Gas Recipe

Pak 'n' Save's AI-based app, Savey Meal-bot, inadvertently suggested dangerous recipes, including one creating chlorine gas, when users entered non-food household items, raising safety and oversight concerns.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI system failed to perform reliably when presented with non-food ingredients, lacking the robustness to filter out hazardous household chemicals and instead generating dangerous recipes.

Additional risk subdomains

• 1.2 Exposure to toxic content: The bot exposed users to highly unsafe and harmful content by describing and recommending the creation of deadly chlorine gas and poison sandwiches.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk was caused by the Savey Meal-bot AI system generating hazardous recipes post-deployment, which was an unexpected and unintentional outcome of its goal to generate meal plans.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a chatbot ('Savey Meal-bot') that generates content and interacts with users, which falls under Risk Level 3 requiring transparency obligations so users know they are interacting with an AI.

AI system and alleged parties

• AI system: Savey Meal-bot
• AI purpose: Chatbot; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: Pak 'n' Save
• Alleged deployer: Pak 'n' Save
• Alleged harmed parties: Potential users of the Savey Meal-bot

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the report explicitly describes toxic and unsafe content created by the AI system.

Directly caused: The AI system directly generated recipes for deadly chlorine gas, bleach mocktails, and poison sandwiches, which were then shared on social media platforms like Twitter.

Epistemic

Reported: Yes, the report describes the AI system generating false and misleading information regarding the safety of chemical mixtures.

Directly caused: The AI system falsely described a recipe that would create deadly chlorine gas as 'the perfect nonalcoholic beverage to quench your thirst and refresh your senses'.

People affected

• Occurrences reported: 1
• People reportedly exposed: 10

Potential causes

Management

• Transfer of Safety Risk: Management shifted safety responsibility to users via disclaimers.
• Inadequate Risk Assessment: Failed to foresee risks of users entering non-food household items.

Technology

• No Input Filtering: App accepted hazardous non-food items like bleach and glue as ingredients.
• Unconstrained LLM Generation: The AI model lacked safety guardrails to block hazardous recipe generation.

Data Inputs

• Harmful User Inputs: Users entered toxic household chemicals into the recipe generator.
• Lack of Ingredient Validation: The system did not validate if inputs were safe, edible food items.

Human Factors

• Adversarial User Behavior: Users actively experimented with toxic inputs to test the AI's limits.
• Overreliance on Disclaimers: Management assumed terms and conditions would prevent unsafe user actions.

Process and Methods

• No Human in the Loop: Recipes were generated and displayed instantly without human review.
• Inadequate Safety Testing: The application was deployed without testing against adversarial inputs.

Regulatory Environment

• Lack of AI Safety Standards: No regulatory mandates enforced safety guardrails on consumer AI apps.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly details the AI system, its developer, the specific outputs generated, and the public reaction. There is no ambiguity about what the AI did or how users interacted with it.

A supermarket meal-planning AI generated hazardous recipes, including chlorine gas, after users entered non-food items. While presenting a localized public safety concern, the incident did not compromise physical infrastructure, government functions, or national security, representing a commercial AI safety failure rather than a national security threat.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: New Zealand
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing industry-wide challenge with AI guardrails rather than an active national security crisis.
• Autonomy: Human-supervised. The AI generated recipes autonomously, but required human action to input ingredients and physically create the mixtures.
• Novelty: Established threat. AI systems generating hazardous or toxic instructions due to poor input filtering is a known and established vulnerability.

Impact by dimension

• Physical security: Negligible. No physical injuries, poisonings, or damage to critical infrastructure were reported. The incident involved a localized consumer app error.
• Information security: Negligible. No evidence of state-sponsored disinformation, intelligence compromise, or systematic information warfare.
• Sovereignty: Negligible. The incident did not impact government operations, state sovereignty, or electoral systems.
• Economic security: Negligible. No strategic economic systems, critical supply chains, or national technological advantages were compromised.
• Societal stability: Negligible. While the bot generated hazardous content shared on social media, it did not cause societal instability, civil unrest, or human rights violations.