Facebook's Automated Tools Failed to Adequately Remove Hate Speech, Violence, and Incitement

Facebook's automated moderation tools were shown by internal documents performing incomparably to human moderators, and accounting for only a small fraction of hate speech, violence, and incitement content removal.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The primary failure of the AI moderation system resulted in widespread exposure of users to toxic content, including hate speech, violence, incitement, and mass shooting videos.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI system demonstrated a severe lack of capability and robustness, misclassifying mass shootings as paintball and failing to generalize across different languages.
• 1.3 Unequal performance across groups: The AI system performed unequally across linguistic groups, with a near-total failure to detect hate speech in Afghanistan (0.23%) compared to English-speaking regions.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk of hate speech and violence slipping through was caused by the failure of the deployed AI moderation tools to perform robustly, which was an unintentional outcome of the system's technical limitations.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The automated moderation tools function similarly to spam filters, which are classified as posing minimal or no risk under the EU AI Act.

AI system and alleged parties

• AI system: Facebook's automated moderation tools (Meta)
• AI purpose: Content Moderation; Hate Speech Detection
• Behaviour type: Tool
• Alleged developer: Facebook
• Alleged deployer: Facebook
• Alleged harmed parties: Facebook users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Substantial, indirect Substantial
• Differential treatment: direct Substantial, indirect Negligible
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes the widespread dissemination of hate speech, violence, incitement, and mass shooting videos.

Directly caused: The AI system failed to remove toxic content, allowing 95-97% of hate speech views and 99.4% of violence/incitement views to persist.

Indirectly caused: The failure of the AI tools allowed toxic content to remain accessible, facilitating its spread across the platform.

Inferred additional harm: We infer that billions of views of toxic content occurred globally due to the AI's low detection rates, particularly in non-English speaking countries.

Differential treatment

Reported: The report explicitly describes unequal performance of the moderation tools across different language groups.

Directly caused: The AI system performed significantly worse for non-English speakers, detecting only 0.23% of hate speech in Afghanistan due to a lack of a dictionary of local slurs.

Inferred additional harm: We infer that users in non-English speaking regions suffered disproportionate exposure to harmful content due to the AI's linguistic limitations.

Civil rights

Reported: The report mentions concerns from civil rights groups like Color of Change regarding the lack of transparency and the persistence of hate speech.

Indirectly caused: The failure to moderate hate speech undermines the civil rights of targeted minority groups who use the platform.

Inferred additional harm: We infer that systemic exposure to unmoderated hate speech contributed to the erosion of civil rights and safety for marginalized communities globally.

Psychological

Reported: The report does not explicitly quantify psychological harm, but notes that users were exposed to hate speech and mass shooting videos.

Indirectly caused: The failure of the AI tools allowed graphic violence and hate speech to remain on the platform, causing distress to users who encountered them.

Inferred additional harm: We infer that millions of users globally experienced psychological distress, anxiety, or trauma due to encountering unmoderated hate speech and graphic violence.

People affected

• Occurrences reported: 1
• People reportedly harmed: 100000000
• People reportedly exposed: 2900000000

Potential causes

Management

• Prioritizing Cost Over Safety: Targeted a 15 percent reduction in human content moderation review costs.
• Misleading Public Communication: Management refused to disclose metric calculations to external groups.
• Focus on Avoiding Take-downs: Leadership forced engineers to prioritize avoiding false positives over safety.

Technology

• Inability to Understand Context: AI confused cockfights with car crashes and mass shootings with paintball.
• Low Recall of Integrity Harms: AI tools only caught 3-5 percent of views of hate speech on the platform.
• Missing Local Language Support: Lack of dictionaries in languages like Dari and Pashto limited detection.

Data Inputs

• Lack of Local Slur Dictionaries: No dictionaries for local Afghan languages hindered training data.
• Flawed Evaluation Datasets: Estimations relied on small samples of posts to guess platform-wide harms.

Human Factors

• User Reporting Barriers: Making it harder for users to report content falsely boosted AI performance.
• User Safety Preferences Ignored: Users favored aggressive enforcement, but leadership ignored this preference.

Process and Methods

• Ignoring User Reports: Algorithms were tweaked to ignore user reports to lower human review costs.
• Misleading Performance Metrics: Using proactive detection rate to obscure low platform-wide recall.
• Optimizing to Avoid False Positives: Training models to avoid false positives allowed hate speech to slip through.

Information quality

• Classification confidence: High
• Reason for confidence: The report is highly detailed, drawing from leaked internal Facebook documents and whistleblower testimony, providing specific statistics on the AI's failure rates (e.g., removing only 3-5% of hate speech views) and organizational context.
• Ambiguities identified: There is some ambiguity regarding the exact technical architecture of the AI models and how the 'proactive detection' metrics publicly reported (98%) are calculated versus the internal 'views' metrics.
• Alternative interpretations: One could interpret the incident primarily as a corporate governance and public relations failure rather than a purely technical AI failure, given that management intentionally made user reporting harder to manipulate metrics.

Internal leaks revealed that Facebook's AI moderation tools failed to remove 95-97% of hate speech and 99.4% of violence/incitement, with even worse performance in non-English regions like Afghanistan. While not a direct kinetic or state-sponsored threat, this systemic failure represents a substantial threat to global societal stability and human rights by permitting the unchecked spread of toxic content.

• Overall national security impact: Substantial
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global, Afghanistan
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing, systemic platform vulnerability and policy challenge rather than an immediate, acute national security crisis.
• Autonomy: Human-supervised. The AI moderation tools make automated classification decisions, but operate alongside human review processes and management oversight.
• Novelty: Established threat. The technical limitations of automated content moderation and NLP tools in capturing nuance and local slurs are well-documented, persistent challenges.

Impact by dimension

• Physical security: Minor. Indirect risks to physical safety arose from the failure to suppress violent incitement and mass shooting footage, though no direct AI-driven kinetic attacks occurred.
• Information security: Minor. The failure facilitated the spread of harmful content and potential influence operations, though the AI system itself was not an active adversarial information warfare tool.
• Sovereignty: Negligible. Minimal direct impact on core government operations or state sovereignty, though systemic platform instability can indirectly affect political environments.
• Economic security: Minor. Limited to corporate governance and cost-cutting decisions by Facebook, with no significant threats to national economic systems or strategic technologies.
• Societal stability: Substantial. Notable impact. Systemic failure to moderate hate speech and violence on a global scale eroded social cohesion and human rights, especially in volatile, non-English speaking regions like Afghanistan.