Facebook’s Algorithm Mistook an Advertisement of Onions as Sexual Suggestive Content

Facebook’s content moderation algorithm misidentified and removed a Canadian business’s advertisement containing a photo of onions as products of overtly sexual content, which was later reinstated after review.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI system failed to perform reliably under normal conditions, misidentifying a benign image of onions as sexually suggestive content due to a lack of robustness in its computer vision capabilities.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by an unexpected false positive decision made by Facebook's automated content moderation AI after it was deployed.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The AI system is an automated content moderation filter used to keep nudity off social media apps, which poses minimal risk to users and is analogous to simple applications like spam filters.

AI system and alleged parties

• AI system: Facebook content moderation algorithm (Meta)
• AI purpose: Content Moderation; NSFW Content Detection
• Behaviour type: Autonomous
• Alleged developer: Facebook
• Alleged deployer: Facebook
• Alleged harmed parties: The Seed Company by E.W. Gaze, businesses on Facebook

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Reliance on Automated Systems: High volume of moderation (35.7M removals) handled mostly by AI.

Technology

• Errant Moderation Algorithm: AI failed to distinguish round onions from human nudity.
• Over-zealous Computer Vision: Automated tech flagged non-sexual round shapes as suggestive.

Data Inputs

• Ambiguous Visual Features: The shape of Walla Walla onions resembled human anatomy to the AI.

Human Factors

• Lack of Immediate Human Review: No human verified the automated ban before it was enacted.

Process and Methods

• Automated First-Pass Moderation: System automatically takes down flagged content without human check.
• Delayed Appeal Process: No immediate reply or quick path to human review after flagging.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly consistent, clear, and explicitly state that an automated algorithm incorrectly flagged an onion advertisement as sexual. The details of the system's action, the company's reaction, and Facebook's response and apology are well-documented across multiple sources.

A Facebook content moderation AI temporarily blocked an advertisement for onion seeds after misinterpreting the round shape of the onions as sexually suggestive. The incident was a minor technical error with zero national security implications, resolved quickly through human review.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: Canada
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident is resolved and represents a minor, non-threatening software error with no ongoing or impending national security threat.
• Autonomy: Human-supervised. The AI system operated autonomously to flag the content, but was subject to human review and override upon appeal.
• Novelty: Established threat. False positives in automated content moderation systems are well-documented, common occurrences rather than a novel capability or threat.

Impact by dimension

• Physical security: Negligible. The incident involves a false positive in a social media content moderation algorithm flagging onion seeds, presenting zero threat to physical systems or infrastructure.
• Information security: Negligible. While involving automated moderation, the incident is a benign false positive on a local commercial advertisement, with no connection to information warfare or intelligence compromise.
• Sovereignty: Negligible. The incident is a localized commercial advertising error on a private social media platform and has no impact on state authority, electoral processes, or government operations.
• Economic security: Negligible. The temporary suspension of a local business advertisement caused no systemic economic harm or strategic technological loss, and actually resulted in positive publicity for the company.
• Societal stability: Negligible. The automated flagging of an onion image as suggestive was a minor technical error with no implications for civil liberties, human rights, or societal stability.