TikTok’s Content Moderation Allegedly Failed to Adequately Take down Videos Promoting Eating Disorders

Videos promoting eating disorders evaded TikTok's automated violation detection system without difficulty via common misspellings of search terms, bypassing its ban of violating hashtags such as "proana" and "anorexia".

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The TikTok recommendation algorithm exposed users to harmful and unsafe pro-eating disorder content on their For You Pages, which constitutes exposure to toxic content.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The platform's search filters and content moderation systems lacked the robustness to prevent users from bypassing hashtag bans using simple misspellings.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The recommendation algorithm unintentionally distributed harmful eating disorder content to users post-deployment as an unexpected outcome of optimizing for engagement.

EU AI Act risk tier

• Risk tier: 1 Unacceptable

Unacceptable Risk: The report describes an algorithm that creates a 'vicious cycle' of behavioral influence, trapping vulnerable young users by continuously showing them harmful eating disorder content, which aligns with systems causing behavioral influence leading to harm.

AI system and alleged parties

• AI system: TikTok For You Page (FYP) algorithm, TikTok's automated violation detection system
• AI purpose: Content Recommendation; Automated Content Curation
• Behaviour type: Autonomous
• Alleged developer: TikTok
• Alleged deployer: TikTok
• Alleged harmed parties: TikTok users under 18 years old, TikTok users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Minor
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the report explicitly describes the spread of pro-anorexia and pro-bulimia content on the platform.

Directly caused: The TikTok recommendation algorithm actively distributed pro-eating disorder content to users' For You Pages, even if they did not search for it.

Indirectly caused: The algorithm's engagement-based design allowed pro-eating disorder content to go viral, with the hashtag 'proana' receiving 2.1 million views and 'anorexia' receiving 446,000 views.

Inferred additional harm: It is likely that millions of additional views of toxic eating disorder content occurred through bypassed search terms and misspellings that went undetected by the platform's filters.

People affected

• Occurrences reported: 1
• People reportedly exposed: 2100000

Potential causes

Management

• Reactive Moderation Strategy: Banning accounts only after external investigations highlight issues.

Technology

• Recommendation Loop: FYP algorithm traps users in cycles of harmful eating disorder content.
• Bypass via Search Terms: Users bypass hashtag bans using plain text search terms.
• Circumvention via Misspellings: Users bypass keyword blocks by using common misspellings of banned words.

Data Inputs

• Harmful Content Uploads: Pro-anorexia and pro-bulimia content uploaded directly by users.
• Unlabeled User Metadata: Lack of precise metadata labels on harmful videos hinders filtering.

Human Factors

• Vulnerable Target Audience: Large share of young users (16-24) susceptible to harmful body trends.
• Active Search Circumvention: Users actively search for harmful terms to bypass content restrictions.

Process and Methods

• Incomplete Filtering Rules: Restricting hashtags but leaving plain text search completely open.
• Lack of Proactive Detection: Absence of automated tools to recognize and downrank self-harm content.

Regulatory Environment

• Lack of Algorithmic Standards: No external regulations governing algorithmic content recommendation safety.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear details about how the TikTok recommendation algorithm functions, the specific hashtags involved, and the platform's response. The role of the AI is explicitly defined as the primary driver of content distribution.
• Ambiguities identified: The exact number of users who developed eating disorders directly due to the algorithm is not quantified, and the technical details of the bypass mechanisms are only briefly described.
• Alternative interpretations: The incident could be viewed purely as a content moderation failure by human staff rather than an algorithmic failure, though the algorithm's active recommendation role makes the AI-centric interpretation highly compelling.

TikTok's recommendation algorithm unintentionally distributed harmful eating disorder content to vulnerable young users due to engagement-based optimization. While raising significant public safety and corporate responsibility concerns, the incident has negligible direct national security implications.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing algorithmic design and content moderation challenge rather than an immediate national security crisis.
• Autonomy: Full autonomy. The recommendation algorithm independently selects and distributes content based on engagement metrics without manual human intervention.
• Novelty: Established threat. Algorithmic amplification of extreme content and simple bypass methods like keyword misspellings are well-documented, persistent issues.

Impact by dimension

• Physical security: Negligible. No impact on physical systems, critical infrastructure, or kinetic safety capabilities.
• Information security: Negligible. No indication of coordinated foreign influence, espionage, or systematic information warfare targeting national intelligence.
• Sovereignty: Negligible. No disruption to state authority, government decision-making, or electoral systems.
• Economic security: Negligible. No direct threat to strategic industries, financial stability, or technological competitive advantage.
• Societal stability: Minor. Exposed a large demographic of young citizens to harmful content, posing minor public health concerns, but lacks direct national security implications.