Facebook's Automated Moderation Flagged Gardening Group's Language Use by Mistake

Facebook's automated system flagged gardening groups' use of "hoe" and violent language against bugs as a violation by mistake.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI system failed to perform reliably under normal conditions due to a lack of capability to understand linguistic context, leading to false-positive flags on benign words like 'hoe' and pest-control discussions.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by Facebook's deployed automated content moderation algorithm incorrectly flagging benign gardening terms and discussions as violations of community standards.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The AI system is used for automated content moderation, which is a low-risk application similar to spam filters, posing negligible risk to safety or fundamental rights.

AI system and alleged parties

• AI system: Facebook's automated system (Meta)
• AI purpose: Content Moderation; Hate Speech Detection
• Behaviour type: Autonomous
• Alleged developer: Facebook
• Alleged deployer: Facebook
• Alleged harmed parties: WNY Gardeners, gardening Facebook groups, Facebook users in gardening groups

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 2
• People reportedly harmed: 1
• People reportedly exposed: 7500

Potential causes

Management

• Neglect of Customer Support: Facebook failed to build adequate support and policy enforcement transparency.

Technology

• Context-Blind NLP Algorithms: AI lacks understanding of linguistic nuance and context like gardening terms.
• Overly Zealous Automation: Automated systems flag benign words like hoe without human verification.

Data Inputs

• Homophone Confusions: Algorithms confuse the tool hoe with a phonetically identical slur.

Human Factors

• User Frustration with Appeals: Group admins find navigating automated appeal surveys futile and useless.

Process and Methods

• Flawed Automated Flagging Process: System flags posts and threatens group deletion without human review.
• Inadequate Support Channels: Lack of accessible human-in-the-loop support for system errors.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly outlines the events, the AI system's role in flagging benign words, the impact on the gardening group, and Facebook's response. There is little ambiguity about what transpired.

An automated content moderation failure on Facebook incorrectly flagged benign gardening terms as policy violations. This incident represents a minor, non-malicious AI safety failure with negligible implications for national security.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This represents an ongoing, long-term technical limitation of NLP models rather than an active national security crisis.
• Autonomy: Full autonomy. The automated moderation system flagged posts, deleted comments, and disabled commenting autonomously without human intervention.
• Novelty: Established threat. Linguistic misinterpretation and false positives in automated content moderation are well-established technical limitations.

Impact by dimension

• Physical security: Negligible. The incident involves automated content moderation of a local gardening group with no physical systems, infrastructure, or human safety implications.
• Information security: Negligible. This is a routine content moderation error affecting a hobbyist group, with no information warfare, disinformation, or intelligence implications.
• Sovereignty: Negligible. No sovereign functions, government processes, or electoral systems were disrupted or targeted by this localized algorithmic error.
• Economic security: Negligible. Minimal economic or technological security impact, restricted entirely to a local social media group's communication.
• Societal stability: Negligible. The incident caused minor, temporary frustration for a local gardening group's members but did not impact societal stability or constitute systematic human rights violations.