Facebook, Instagram, and Twitter Failed to Proactively Remove Targeted Racist Remarks via Automated Systems

Facebook's, Instagram's, and Twitter's automated content moderation failed to proactively remove racist remarks and posts directing at Black football players after finals loss, allegedly largely relying on user reports of harassment.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The AI system's failure to detect and filter racist emojis directly exposed the targeted players and the public to toxic hate speech and harassment.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI lacked the capability and contextual robustness to correctly identify emojis being used as racist markers.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the failure of Instagram's deployed content moderation AI, which unintentionally misclassified racist emoji abuse as benign comments.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The AI system is a content moderation filter, which is functionally similar to a spam filter and poses low or negligible risk to society under the EU AI Act definitions.

AI system and alleged parties

• AI purpose: Hate Speech Detection; Content Moderation
• Behaviour type: Assistant
• Alleged developer: Twitter, Instagram, Facebook
• Alleged deployer: Twitter, Instagram, Facebook
• Alleged harmed parties: Twitter Users, Marcus Rashford, Jadon Sancho, Instagram users, Facebook users, Bukayo Saka

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes hundreds of racist remarks and abusive emojis spread on the platform.

Directly caused: The AI's failure allowed hundreds of toxic, racist comments containing monkey and banana emojis to remain visible on the players' public profiles.

Inferred additional harm: Millions of social media users who visited the players' profiles were exposed to this toxic hate speech before it was manually removed.

Differential treatment

Reported: The report explicitly describes racist abuse targeted specifically at Black players based on their race.

Directly caused: The AI's failure to detect emoji-based racism resulted in Black players being subjected to unchecked racial harassment compared to their white teammates.

Civil rights

Reported: The report explicitly describes attacks based on protected characteristics including race.

Directly caused: The players' civil right to be free from racial discrimination and harassment was violated on the platform due to the moderation failure.

Psychological

Reported: The report explicitly describes targeted racist abuse and harassment directed at the players.

Directly caused: The three targeted Black players experienced severe public racist abuse and harassment due to the AI's failure to filter the comments.

Inferred additional harm: The targeted players likely experienced significant emotional distress, anxiety, and psychological trauma from the high volume of public racial hostility.

People affected

• Occurrences reported: 1
• People reportedly harmed: 3
• People reportedly exposed: 3

Potential causes

Management

• Acceptance of Scale Gaps: Accepting a low error percentage resulted in thousands of missed abuses.
• Inadequate Proactive Controls: Management failed to implement proactive filters for known emoji threats.

Technology

• Flawed Contextual AI: AI failed to understand the racist context of specific emoji combinations.
• Mistaken Benign Classifications: AI mistakenly classified offensive emoji comments as benign.

Data Inputs

• Lack of Emoji Abuse Data: System lacked training data on emojis used as racial markers.

Human Factors

• Exploitative User Behavior: Trolls used emojis to evade standard text-based moderation filters.
• Onus Placed on Victims: Victims and fans had to manually report abuse to trigger reviews.

Process and Methods

• Reactive Moderation Model: The platform relied on reports instead of proactive emoji filtering.
• Delayed Review Prioritization: AI misclassification delayed human review of critical emoji reports.

Regulatory Environment

• Lack of Strict Mandates: No regulatory penalties existed to enforce immediate removal of abuse.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, consistent details about the failure of Instagram's AI to detect racist emojis, the specific individuals targeted, and the response from Facebook executives and public figures. There is little ambiguity about the nature of the failure or the system involved.

Following the Euro 2020 final, Instagram's automated content moderation AI failed to detect and block racist emoji-based abuse targeting Black English football players. While the incident caused significant public distress, political debate, and highlighted moderation vulnerabilities, its direct national security impact remains minor and limited to societal and regulatory discussions.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United Kingdom
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing, systemic challenge in algorithmic content moderation rather than an active national security crisis.
• Autonomy: Human-supervised. The AI operated autonomously to filter comments, but relied on human reporting and manual reviews to correct its classification errors.
• Novelty: Established threat. Algorithmic failure to detect nuanced or contextual hate speech, including emojis, is a well-documented and established technical limitation.

Impact by dimension

• Physical security: Negligible. No physical threat, kinetic action, or critical infrastructure disruption was associated with this content moderation failure.
• Information security: Negligible. The incident involved uncoordinated online harassment by sports fans rather than a state-sponsored information warfare campaign or intelligence breach.
• Sovereignty: Negligible. Core government functions, territorial sovereignty, and democratic processes remained unaffected, though it prompted political calls for regulation.
• Economic security: Negligible. There was no compromise of strategic technologies, critical supply chains, or national economic stability.
• Societal stability: Minor. The AI's failure exposed public figures to unchecked racial harassment, causing public outcry and highlighting systemic challenges in protecting human rights online.