Alexa Recommended Dangerous TikTok Challenge to Ten-Year-Old Girl

Amazon’s voice assistant Alexa suggested “the penny challenge,” which involves dangerously touching a coin to the prongs of a half-exposed plug, when a ten-year-old girl asked for a challenge to do.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The AI systems exposed minor users to highly toxic and dangerous content, including a life-threatening physical challenge and videos promoting self-harm, suicide, and eating disorders.

Additional risk subdomains

• 7.3 Lack of capability or robustness: Amazon's Alexa failed to perform robustly by pulling a highly dangerous physical challenge from the web and recommending it to a child.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incidents were caused by deployed AI systems (Alexa and TikTok's recommendation algorithm) generating unexpected and harmful outputs post-deployment.

EU AI Act risk tier

• Risk tier: 1 Unacceptable

Unacceptable Risk. The report describes TikTok's recommendation algorithm as an exploitative system that targets the vulnerabilities of teenagers based on their age and mental state, bombarding them with harmful self-harm and suicide content.

AI system and alleged parties

• AI system: Alexa (Amazon)
• AI purpose: AI Voice Assistant; Content Recommendation
• Behaviour type: Assistant
• Alleged developer: Amazon
• Alleged deployer: Amazon
• Alleged harmed parties: Kristin Livdahl's daughter, children using Alexa, Amazon Echo customers

Harm severity

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Substantial, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and malicious content being spread by both AI systems.

Directly caused: Amazon's Alexa directly outputted a dangerous challenge instructing a child to touch a penny to live electrical prongs. TikTok's algorithm directly recommended videos promoting suicide, self-harm, and extreme eating disorders to teenage accounts.

Indirectly caused: N/A

Inferred additional harm: It is highly likely that millions of other users were exposed to similar toxic recommendations across both platforms.

Differential treatment

Reported: The report explicitly describes differential treatment of vulnerable users by TikTok's algorithm.

Directly caused: TikTok's algorithm targeted 'vulnerable' accounts (with 'loseweight' in the username) with three times more body image/mental health content and twelve times more self-harm/suicide recommendations than standard accounts.

Indirectly caused: N/A

Inferred additional harm: Vulnerable teenagers seeking help or expressing insecurities are systematically targeted with more extreme and harmful content by the recommendation engine.

People affected

• Occurrences reported: 2
• People reportedly exposed: 1

Potential causes

Management

• Reactive Safety Mitigation: Action was taken only after public exposure of the safety incidents.
• Prioritizing Platform Growth: Focus on user engagement led to insufficient safety oversight.

Technology

• Unfiltered Web Scraping: Assistant pulled dangerous instructions directly from the web.
• Engagement-Driven Algorithms: Recommendation engine prioritized watch-time over user safety.
• Lack of Real-time Safety Guardrails: System failed to block hazardous physical challenge recommendations.

Data Inputs

• Raw Search Index Data: System used unvetted web source data for answering user queries.
• User Behavioral Signals: Brief pauses and likes on videos triggered harmful recommendations.

Human Factors

• User Curiosity and Vulnerability: Minor users sought out challenges and self-harm content online.
• Lack of Parental Supervision: Children interacted with AI devices and apps unsupervised.

Process and Methods

• Inadequate Content Filtering: Platforms failed to proactively block known dangerous challenges.
• Deficient Safety Testing: Voice assistant was not properly tested against unsafe web content.

Regulatory Environment

• Weak Online Safety Regulations: Absence of strict legal mandates to prevent exposure of minors to harm.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, detailed accounts of both the Alexa incident and the CCDH study on TikTok. The facts are straightforward, and the roles of the AI systems are explicitly defined.
• Alternative interpretations: The TikTok algorithm's behavior could be interpreted as a design optimization for engagement rather than intentional exploitation, though the outcome remains highly harmful.

The incidents involve consumer AI systems exposing minors to physical and psychological risks, specifically Amazon Alexa suggesting a dangerous physical challenge and TikTok's algorithm systematically promoting self-harm content to vulnerable teens. While primarily public safety issues, the scale of TikTok's algorithmic impact on Western youth represents a substantial societal stability concern.

• Overall national security impact: Substantial
• Response level: Substantial
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: United States, United Kingdom, Canada, Australia
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing algorithmic recommendation capability and systemic public safety concern rather than an immediate, acute national security crisis.
• Autonomy: Full autonomy. The AI recommendation engines and voice assistants generate outputs and curate content feeds autonomously without direct human intervention.
• Novelty: Evolved capability. While algorithmic recommendation is an established technology, the specific targeting of vulnerable minor accounts with highly toxic content represents an evolved threat scale.

Impact by dimension

• Physical security: Negligible. No significant threat to critical infrastructure or physical systems. Alexa's dangerous suggestion was intercepted by a parent without physical harm occurring.
• Information security: Minor. Algorithmic promotion of self-harm to minors represents a public health concern but does not constitute an active information warfare or intelligence compromise campaign based on the provided text.
• Sovereignty: Negligible. No threats to state authority, territorial control, or core government functions were indicated in the incident details.
• Economic security: Negligible. No significant threats to economic stability, strategic industries, or technological competitive advantage are present in these consumer-focused incidents.
• Societal stability: Substantial. TikTok's systematic algorithmic promotion of self-harm and suicide content to minors across multiple Western nations represents a substantial threat to population-scale societal stability and public safety.