Facebook AI-Supported Moderation for Ads Failed to Detect Violating Content

Facebook's ad moderation system involving algorithms failed to flag hateful language and violating content such as calls for killings for ads in English and Swahili.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The AI system approved advertisements containing violent hate speech, dehumanizing language, and calls for murder, creating a severe risk of exposing users to toxic content.

Additional risk subdomains

• 1.3 Unequal performance across groups: The moderation system performed significantly worse for Swahili and other regional languages compared to English, resulting in unequal safety outcomes for different language groups.
• 7.3 Lack of capability or robustness: The AI system demonstrated a fundamental lack of capability and robustness in detecting obvious hate speech and violent content in multiple languages.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the failure of Meta's deployed automated ad moderation AI to detect policy-violating content, which was an unintended outcome of its operation.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The system is an automated content filter, which is analogous to a spam filter, classified under Level 4 as it does not fall under the high-risk or prohibited categories defined in the Act.

AI system and alleged parties

• AI system: Facebook ad moderation system (Meta)
• AI purpose: Content Moderation; Hate Speech Detection
• Behaviour type: Tool
• Alleged developer: Facebook
• Alleged deployer: Facebook
• Alleged harmed parties: Facebook users speaking Swahili, Facebook users speaking English, Facebook users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes the creation and approval of advertisements containing violent hate speech and calls for murder.

Directly caused: The AI system approved violent hate speech ads in English and Swahili, as well as dehumanizing ads in Myanmar and Ethiopia, though they were not published.

Inferred additional harm: It is highly likely that similar actual hate speech ads and posts regularly bypass the system and are spread to millions of users, given the 100% failure rate in the tests.

Differential treatment

Reported: The report explicitly describes differential treatment of language groups, as Swahili ads bypassed filters more easily than English ads.

Directly caused: The AI system approved Swahili ads immediately while initially flagging some English ads, showing differential performance based on language.

Inferred additional harm: Non-English speaking populations in East Africa, Myanmar, and Ethiopia face a disproportionately higher risk of exposure to violent hate speech due to the system's language bias.

People affected

• Occurrences reported: 3

Potential causes

Management

• Prioritizing Growth Over Safety: Resource allocation favored expansion over robust regional safety.
• Lack of Operational Transparency: Refusal to disclose moderator numbers hid gaps in content safety.
• Inadequate Risk Mitigation: Management failed to secure the platform ahead of the election.

Technology

• Flawed AI Detection Algorithms: AI models failed to identify explicit hate speech and violent content.
• Inability to Parse Swahili Language: Detection technology struggled to process Swahili language nuances.
• Deficient English Grammar Filters: System flagged grammar/profanity but missed underlying hate speech.

Data Inputs

• Inadequate Training Data: AI lacked exposure to diverse regional dialects and hate speech patterns.
• Lack of Regional Context Data: System lacked context on local ethnic tensions and derogatory terms.

Human Factors

• Inadequate Human Moderation: Human moderators failed to catch blatant hate speech in approved ads.
• Lack of Local Language Moderators: Insufficient Swahili and regional language speakers to review content.

Process and Methods

• Flawed Ad Review Workflow: System approved ads once superficial grammar issues were resolved.
• Ineffective Testing Procedures: Internal testing failed to identify gaps in hate speech detection.

Regulatory Environment

• Lack of Regulatory Oversight: Absence of strong external regulations allowed safety gaps to persist.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, direct evidence of the tests conducted by Global Witness and Foxglove, including the specific languages used, the nature of the content, and the outcomes of the submissions. There is minimal ambiguity regarding the system's failure to block the test ads.
• Ambiguities identified: The exact ratio of AI vs human moderator failure in these specific approvals is not fully disentangled, though the report focuses on the failure of the automated detection systems.

Nonprofit groups exposed significant vulnerabilities in Meta's AI-supported ad moderation, which approved violent hate speech and calls for murder in English and Swahili. Because the test ads were never published, the direct national security impact is minor, but the systemic failure highlights ongoing risks to societal stability and democratic processes in volatile regions.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Kenya, Myanmar, Ethiopia
• Alleged perpetrator: Global Witness and Foxglove

Threat characteristics

• Imminence: Long-term. The test ads were not published, meaning there is no immediate crisis, representing an ongoing strategic concern regarding content moderation in non-English languages.
• Autonomy: Human-supervised. The ad moderation system relies on automated AI detection with some human oversight and manual review processes, though the automated layer failed.
• Novelty: Established threat. Vulnerabilities in automated content moderation systems and failure to detect non-English hate speech are well-documented issues that have occurred in previous conflicts.

Impact by dimension

• Physical security: Negligible. The test advertisements containing violent calls to action were never published to the live platform, resulting in zero direct physical harm or critical infrastructure disruption.
• Information security: Minor. The incident demonstrates a vulnerability where automated systems fail to filter hate speech, though no actual foreign information warfare or intelligence compromise occurred.
• Sovereignty: Minor. The moderation failures occurred ahead of a sensitive national election in Kenya, highlighting a risk to democratic processes, though no actual disruption was executed.
• Economic security: Negligible. No threats to economic stability, financial systems, or strategic technological competitive advantages were reported in this incident.
• Societal stability: Minor. The AI failed to detect severe hate speech and calls for murder in Swahili and English. While the test ads were not published, the vulnerability poses a risk to societal stability.