Discord's Automated Moderation System Reportedly Wrongfully Banned More Than 8,000 Users Over Benign Images

Discord reportedly wrongfully banned more than 8,000 accounts after its automated image-moderation workflow flagged harmless uploads, including grids, spreadsheets, chessboards, and game images, as prohibited material. Discord said a bug converted temporary upload restrictions into account bans and prevented cleared accounts from being restored automatically. The company later reinstated the affected accounts.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The automated safety system failed to perform reliably, incorrectly flagging benign grid-like images as harmful and executing erroneous bans due to a software glitch.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was triggered by an operational AI-driven content moderation system post-deployment, which unintentionally flagged benign images and incorrectly banned users due to a system bug.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The system is an automated content moderation filter, which is analogous to 'spam filters' and other simple applications that pose low or negligible risks to society under the EU AI Act.

AI system and alleged parties

• AI system: None named
• AI purpose: Content Moderation; Image Verification
• Behaviour type: Autonomous
• Alleged developer: Social media platforms, Discord
• Alleged deployer: Social media platforms, Discord
• Alleged harmed parties: Social media users, Discord users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Civil rights

Reported: Yes, the report describes wrongful bans of over 8,000 user accounts, which temporarily restricted their digital access and freedom of association on the platform.

Directly caused: The automated system mistakenly banned over 8,000 users, temporarily depriving them of their accounts and access to the platform.

Inferred additional harm: It is possible that some users experienced temporary disruption to their communication, communities, or digital activities during the period they were wrongfully banned.

People affected

• Occurrences reported: 1
• People reportedly harmed: 8200
• People reportedly exposed: 8200

Potential causes

Technology

• False Positive Matching Glitch: System incorrectly matched benign grid images with known harmful content.
• Immediate Account Ban Bug: Glitch caused immediate account bans instead of temporary upload blocks.
• Failed Ban Reversal Bug: Bug prevented account bans from being automatically lifted after staff review.

Data Inputs

• Grid-like Image Inputs: Chessboards and textures triggered false positives in detection system.
• Benign Image Variations: Other benign images since May triggered false positive matches.

Information quality

• Classification confidence: High
• Reason for confidence: The incident reports from Discord support and tech journalism provide clear, consistent, and explicit details about the nature of the bug, the number of users affected (around 8,200), and the resolution of the incident. There is little ambiguity about what transpired.
• Alternative interpretations: The incident could be interpreted purely as a software engineering bug rather than an AI failure, but since it involves an automated image-matching safety system, it is classified as an AI robustness failure.

An unintentional software bug in Discord's automated image-moderation system caused wrongful bans of over 8,000 accounts. The incident was quickly resolved and accounts were reinstated, presenting negligible national security implications across all assessed categories.

• Overall national security impact: Negligible
• Scope: Unknown
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The glitch has been resolved and all accounts reinstated, representing no active crisis.
• Autonomy: Full autonomy. The automated system executed bans directly without human intervention due to a software bug.
• Novelty: Established threat. Software bugs and false positives in automated content moderation systems are well-established and common occurrences.

Impact by dimension

• Physical security: Negligible. No impact on physical systems, critical infrastructure, or human safety.
• Information security: Negligible. No evidence of information warfare, intelligence compromise, or systematic information manipulation.
• Sovereignty: Negligible. No impact on state authority, territorial control, or core government operations.
• Economic security: Negligible. No threat to economic stability, strategic industries, or technological competitive advantage.
• Societal stability: Negligible. Temporary platform access disruption for some users due to a technical glitch, with negligible impact on societal stability or human rights.