Facebook’s Moderation Algorithm Banned Users for Historical Evidence of Slavery

Facebook’s automated content moderation was acknowledged by a company spokesperson to have erroneously censored and banned Australian users from posting an article containing a 1890s photo of Aboriginal men in chains over nudity as historical evidence of slavery in Australia.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI system failed to perform robustly under varying conditions, misclassifying a historical photo of Aboriginal men in loincloths as policy-violating nudity.

Additional risk subdomains

• 1.3 Unequal performance across groups: The automated censorship system disproportionately censored discussions of historical racism and minority history, reflecting unequal impacts across demographic groups.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by an automated content moderation AI system that unintentionally misclassified a historical photo as nudity, leading to unexpected account restrictions.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The system is an automated content moderation filter, which is highly analogous to spam filters and poses low direct risk to physical safety, though it can cause minor civil discourse issues.

AI system and alleged parties

• AI system: Facebook automated content moderation system (Meta)
• AI purpose: Content Moderation; NSFW Content Detection
• Behaviour type: Autonomous
• Alleged developer: Facebook
• Alleged deployer: Facebook
• Alleged harmed parties: Facebook users sharing photo evidence of slavery, Facebook users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Minor, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes concerns regarding the differential impact of automated censorship on minority groups.

Directly caused: The automated system censored a historical photo of Aboriginal men, suppressing discussion of Indigenous history and slavery.

Indirectly caused: Activists noted that the fallout from automated overblocking is not evenly distributed, with marginalized groups like people of color, queer people, and sex workers being more likely to have their discussions censored.

Civil rights

Reported: The report explicitly describes the suppression of political discourse and freedom of expression.

Directly caused: Facebook's automated system blocked users from sharing a historical photo and a news article, restricting their ability to engage in political debate regarding Australia's history of slavery.

Indirectly caused: Users faced account restrictions and bans of up to 30 days, limiting their digital civil liberties and communication capabilities on the platform.

People affected

• Occurrences reported: 1
• People reportedly harmed: 25
• People reportedly exposed: 25

Potential causes

Management

• Over-reliance on Automation: Management deployed automated moderation at scale without sufficient safeguards.
• Resource Prioritization Decisions: Management prioritized high-harm content, leaving standard reviews neglected.

Technology

• Flawed Automated Moderation: Automated system misidentified historical loincloths as violating nudity.
• Inflexible Filtering Algorithms: Algorithms immediately blocked reposts of the news article without context.

Data Inputs

• Lack of Contextual Training Data: The AI lacked the ability to distinguish historical photos from modern nudity.

Human Factors

• Reduced Human Reviewer Workforce: The COVID-19 pandemic reduced the availability of human content moderators.

Process and Methods

• Absence of Immediate Appeal Path: Blocked users were given no immediate option to dispute or appeal the ban.
• Inadequate Incident Escalation: The system repeated the error on reposts until manual intervention occurred.

Regulatory Environment

• Lack of Algorithmic Oversight: No external regulatory standards governed automated content removal decisions.

Information quality

• Classification confidence: High
• Reason for confidence: The reports clearly outline the sequence of events: the automated system flagged a historical photo as nudity, blocked users, and Facebook subsequently apologized and restored the content, attributing it to an automated error. There is minimal ambiguity regarding the AI's role or the immediate consequences.
• Ambiguities identified: The exact algorithm used by Facebook is not detailed, and the precise number of users banned is described generally as 'dozens'.
• Alternative interpretations: None. The incident is clearly an automated content moderation false positive.

An automated content moderation system on Facebook mistakenly flagged and removed a historical photo of Aboriginal men, temporarily suppressing public political discourse in Australia. The incident represents a minor national security concern, highlighting the ongoing challenges of algorithmic bias and automated censorship on private platforms, but poses no threat to physical, sovereign, or economic security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: Australia
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing strategic concern regarding algorithmic bias and automated censorship rather than an active, imminent national security crisis.
• Autonomy: Full autonomy. The content moderation AI operated fully autonomously to flag, remove content, and restrict user accounts without human oversight prior to enforcement.
• Novelty: Established threat. Algorithmic bias, false positives, and overblocking in automated content moderation systems are well-established and frequently documented issues.

Impact by dimension

• Physical security: Negligible. The incident involved automated content moderation on a social media platform and did not pose any threat to physical systems, critical infrastructure, or human safety.
• Information security: Negligible. While the automated system temporarily suppressed political discourse, this was an unintentional platform error rather than a hostile information warfare campaign or intelligence compromise.
• Sovereignty: Negligible. There was no compromise of government decision-making, electoral systems, or state authority, despite the deleted post being part of a debate involving the Prime Minister's statements.
• Economic security: Negligible. No strategic economic assets, financial systems, or technological competitive advantages were threatened or compromised by this content moderation error.
• Societal stability: Minor. The automated overblocking temporarily restricted digital civil liberties and suppressed political discourse regarding indigenous history, demonstrating minor, localized impacts on freedom of expression.