Facebook's Automated Moderation Allowed Ads Threatening Election Workers to be Posted

Facebook's automated moderating system failed to flag and allowed ads containing explicit violent language against election workers to be published.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The automated moderation system failed to detect overt death threats and violent ads, approving 15 out of 20 of them due to a lack of capability and robustness in its natural language processing.

Additional risk subdomains

• 1.2 Exposure to toxic content: The system's failure to block the ads created a direct risk of exposing platform users to extreme violence and death threats.
• 3.1 False or misleading information: The reports highlight a parallel failure where the same automated systems approved election disinformation ads.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk was caused by the automated moderation AI system failing to detect violent ads, which was an unexpected outcome of its deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes an AI system used in a high-stakes context (moderating election-related content) where failures have significant implications for public safety, fundamental rights, and democratic processes.

AI system and alleged parties

• AI system: Facebook's automated moderating system (Meta)
• AI purpose: Content Moderation; Threat Detection
• Behaviour type: Tool
• Alleged developer: Facebook
• Alleged deployer: Facebook
• Alleged harmed parties: Facebook users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Substantial, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Minor, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and violent content in the form of death threats that were approved by the AI system.

Directly caused: The automated moderation system approved 15 advertisements containing explicit death threats and violent language for publication.

Democracy

Reported: The report explicitly describes how the failure to block threats against election workers and election disinformation threatens the safety of elections and democratic processes.

Directly caused: The automated system approved ads containing violent threats against election workers and election disinformation, undermining the integrity of democratic elections.

Inferred additional harm: The systemic failure of moderation algorithms likely allows real voter suppression and intimidation campaigns to run unchecked, eroding trust in democratic institutions.

Epistemic

Reported: The report explicitly describes epistemic harm in the form of election disinformation ads that were approved by the platform's systems in a parallel test.

Directly caused: In a parallel test mentioned in the reports, Facebook approved 20 to 50 percent of election disinformation ads, and TikTok approved 90 percent of them.

People affected

• Occurrences reported: 1

Potential causes

Management

• Under-resourced Content Moderation: Meta failed to adequately resource and fund global safety and integrity systems.
• Overreliance on Automation: Management relied heavily on automated systems without enough human oversight.

Technology

• Flawed Automated Moderation: Facebook's automated system failed to flag explicit death threat text in ads.
• Poor Image-Text OCR Integration: System failed to extract and analyze violent text overlaid on images.
• Language Detection Variance: System performed inconsistently across English and Spanish language ads.

Data Inputs

• Grammar and Profanity Sensitivity: System bypassed checks when profanity was removed and grammar was corrected.
• Text-on-Image Format: Threat text was embedded in images, bypassing text-only classifiers.

Human Factors

• Adversarial Editing of Ads: Researchers bypassed filters by removing profanity and correcting grammar.

Process and Methods

• Inadequate Pre-Election Screening: Moderation processes failed to secure high-risk ads right before elections.
• Lack of Keyword Triggering: Automated system lacked simple keyword flags for manual review of violent terms.
• Post-Live Review Reliance: Facebook relied on post-live reviews instead of robust pre-publication checks.

Regulatory Environment

• Lack of Binding Ad Standards: EU regulation is emerging but global binding standards remain weak or absent.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, empirical results from a structured audit conducted by Global Witness and NYU C4D. The methodology, including the number of ads submitted (20) and approved (15), is explicitly detailed. There is no ambiguity about the AI system's failure to detect the violent content.

A red-teaming investigation revealed that Facebook's automated ad moderation system approved 75% of test ads containing explicit death threats against US election workers. While the ads were deleted before publication to prevent actual harm, the failure highlights a minor but notable vulnerability in information security and election integrity safeguards.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Global Witness and NYU researchers

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding content moderation robustness during democratic events rather than an active, imminent crisis.
• Autonomy: Human-supervised. The AI moderation system autonomously reviews and approves ads, but operates within a platform framework subject to human oversight and administrative control.
• Novelty: Established threat. Algorithmic moderation failures and the spread of election-related disinformation/threats on social media are well-documented, established challenges.

Impact by dimension

• Physical security: Negligible. The test advertisements were deleted before publication, resulting in zero actual physical harm or critical infrastructure disruption.
• Information security: Minor. Demonstrates a platform vulnerability to automated dissemination of violent rhetoric and disinformation, though no active information campaign was executed.
• Sovereignty: Minor. Highlights potential vulnerabilities in safeguarding election workers and processes, but no actual government functions or electoral systems were disrupted.
• Economic security: Negligible. The incident represents a commercial content moderation failure with no impact on strategic economic or technological security.
• Societal stability: Minor. Reveals latent risks of algorithmic failure allowing violent threats against civil servants, though direct societal impact was avoided by deleting the ads.