YouTube's Algorithms Failed to Remove Violating Content Related to Suicide and Self-Harm

Terms-of-service-violating videos related to suicide and self-harm reportedly bypassed YouTube’s content moderation algorithms, allegedly resulting in exposure of graphic content to young users via recommended videos.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The YouTube recommendation algorithm actively surfaced and recommended graphic videos of self-harm and dangerous search terms to vulnerable young users, exposing them to highly inappropriate and harmful content.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The platform's automated moderation and filtering systems failed to reliably detect and block content that violated YouTube's own safety policies.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk was caused by the YouTube recommendation algorithm's post-deployment decisions to surface graphic self-harm videos to young users, which was an unintentional and unexpected outcome of its content recommendation goals.

EU AI Act risk tier

• Risk tier: 1 Unacceptable

Unacceptable Risk: The report describes an AI system used for subliminal manipulation or behavioral influence leading to harm by recommending graphic self-harm videos and tutorials to vulnerable children as young as 13, which exploits their psychological vulnerabilities.

AI system and alleged parties

• AI system: YouTube content moderation algorithms (Google)
• AI purpose: Content Recommendation; Content Curation
• Behaviour type: Autonomous
• Alleged developer: YouTube
• Alleged deployer: YouTube
• Alleged harmed parties: YouTube young users, YouTube users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the report explicitly describes the spread of toxic content, specifically graphic self-harm videos and search recommendations.

Directly caused: The YouTube recommendation algorithm directly promoted graphic self-harm videos and suggested search terms like 'how to self-harm tutorial' and 'self-harming guide' to users.

Inferred additional harm: It is highly likely that many more graphic self-harm videos and dangerous search terms were generated and recommended to millions of users before being flagged and removed.

Psychological

Reported: Yes, the report explicitly describes psychological harm, such as triggering negative emotions and worsening mental health.

Directly caused: The recommendation algorithm directly surfaced triggering self-harm content to a former blogger and her friends, causing them to fall back into bad coping skills and worsening anxiety and depression.

Inferred additional harm: Given that one self-harm video had nearly 400,000 views and search terms like 'how to self-harm tutorial' were recommended, it is highly likely that thousands of vulnerable teenagers experienced increased psychological distress, depression, or self-harm ideation.

People affected

• Occurrences reported: 1
• People reportedly harmed: 5
• People reportedly exposed: 400000

Potential causes

Management

• Prioritizing Engagement Over Safety: Business models optimized for views and watch time over user protection.
• Inadequate Risk Assessment: Failure to anticipate the impact of recommendation engines on minors.

Technology

• Flawed Recommendation Algorithms: Algorithms actively promoted graphic self-harm videos to young users.
• Inadequate Moderation Algorithms: Automated filters failed to detect and block graphic self-harm content.
• Dangerous Autocomplete Suggestions: Search algorithms suggested harmful terms like 'how to self-harm tutorial'.

Data Inputs

• Unlabeled Graphic Uploads: User uploads lacked metadata identifying self-harm content to filters.
• Blurry Support vs Promotion Data: Difficulty distinguishing supportive content from harmful content in data.
• Lack of Proactive Content Tagging: No robust automated tagging system for graphic self-harm images.

Human Factors

• Vulnerability of Young Users: Teenagers fell down rabbit holes of negative content triggering self-harm.
• Irresponsible Content Creation: Creators uploaded graphic self-harm and fake suicide pranks for views.

Process and Methods

• Reactive Flagging System: Platform relied on user flags instead of proactive detection to remove videos.
• Inconsistent Policy Enforcement: Delayed removal of reported videos violating self-harm policies.

Regulatory Environment

• Absence of Statutory Duty of Care: Lack of legal mandates holding tech platforms liable for child safety.
• Weak Regulatory Oversight: No active government enforcement to penalize platforms for harmful content.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from multiple reputable sources consistently describe the algorithm's behavior of recommending graphic self-harm videos and search terms. The role of the recommendation algorithm is explicitly stated, and the resulting public and political reactions are well-documented.
• Ambiguities identified: The exact technical mechanism of the recommendation algorithm's failure is not detailed, and the exact number of children harmed is not quantified.
• Alternative interpretations: One could interpret the issue as primarily a content moderation failure (humans uploading the content) rather than an algorithmic recommendation failure, though the reports emphasize that the algorithm actively recommended the content.

YouTube's recommendation algorithm autonomously promoted graphic self-harm content to minors in the United Kingdom, leading to localized psychological harm and intense regulatory scrutiny. While a significant public safety and child welfare concern, the incident carries negligible to minor direct implications for national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United Kingdom
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing regulatory and public health challenge rather than an active, imminent national security crisis.
• Autonomy: Full autonomy. The recommendation algorithm autonomously selected, ranked, and promoted content to users without real-time human oversight.
• Novelty: Established threat. Algorithmic amplification of policy-violating content on social media platforms is a well-documented and pre-existing phenomenon.

Impact by dimension

• Physical security: Negligible. No threat to physical systems, critical infrastructure, or kinetic national security interests was identified in the incident.
• Information security: Negligible. No evidence of state-sponsored information warfare, intelligence compromise, or systematic foreign disinformation campaigns.
• Sovereignty: Negligible. Core government operations and state sovereignty remained unaffected, though the incident prompted policy discussions regarding child safety.
• Economic security: Negligible. No strategic economic assets, financial systems, or critical technology supply chains were targeted or compromised.
• Societal stability: Minor. The algorithm exposed vulnerable youth to harmful self-harm content, causing localized psychological distress and prompting calls for regulatory reform, but did not threaten overall societal stability.