Wikipedia Vandalism Prevention Bot Loop

Wikipedia bots meant to remove vandalism clash with each other and form feedback loops of repetitve undoing of the other bot's edits.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.6 Multi-agent risks

The incident involves multiple automated bots interacting and clashing in a shared environment, leading to persistent, multi-year edit wars and inefficient resource loops.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The bots failed to perform robustly when encountering other automated agents, falling into infinite loops of reverting edits.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The edit wars were caused by the autonomous decisions of deployed Wikipedia bots reverting each other's edits, which was an unexpected and unintentional outcome of their programming.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The Wikipedia bots are simple automated editing tools used for basic housekeeping tasks like spellchecking and link fixing, which pose low or negligible risks to society.

AI system and alleged parties

• AI system: Darknessbot, Russbot, Scepbot, Xqbot
• AI purpose: Automatic Copy-Editing; Content Moderation
• Behaviour type: Multi-agent
• Alleged developer: Wikipedia
• Alleged deployer: Wikipedia
• Alleged harmed parties: Wikipedia Users, Wikipedia Editors, Wikimedia Foundation

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Inadequate Oversight of Interactions: Management focused on individual bot functions rather than joint interactions.
• Lack of Joint Risk Assessment: Failure to assess risks of deploying multiple independent automated agents.

Technology

• Slightly Different Rulesets: Bots follow different rules, leading to incompatible edits.
• Unpredictable Interaction in Wild: Simple algorithms interact in unexpected, complex ways when deployed.
• Lack of Rational Debate Capability: Bots are servants to code and cannot negotiate or resolve conflicts.

Data Inputs

• Discrepancies in Naming Conventions: Different naming standards for links and redirects trigger mutual reverts.
• Cultural and Language Divergence: Different language editions present differing contexts for bot operations.

Human Factors

• Diverse Developer Backgrounds: Bots designed by developers from different countries leads to clashes.
• Varying Editor Community Cultures: Different language communities create complex, clashing bot interactions.

Process and Methods

• Inadequate Lab Testing: Algorithms working well in labs behave unpredictably when let loose.
• Lack of Inter-Bot Coordination: No standardized coordination protocol exists for bots working on same pages.

Regulatory Environment

• Varying Local Rules and Laws: Different Wikipedia language editions have different local rules and guidelines.
• Lack of Centralized Bot Regulations: No global regulatory framework governs how bots must interact with each other.

Information quality

• Classification confidence: High
• Reason for confidence: The assessment is based on a peer-reviewed scientific study published in PLOS ONE, which provides clear quantitative data on bot-on-bot reverts across multiple language editions of Wikipedia.
• Ambiguities identified: The Wikimedia Foundation disputed whether all reverts represent actual conflict, suggesting some are collaborative multi-step cleanups.
• Alternative interpretations: The interactions could be interpreted as cooperative system maintenance rather than hostile 'edit wars'.

Wikipedia editing bots engaged in persistent edit wars due to conflicting rule sets. While highlighting the challenges of multi-agent AI coordination, the incident has negligible national security implications and poses no threat to critical infrastructure, sovereignty, or societal stability.

• Overall national security impact: Negligible
• Scope: Multiple nations
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents a long-term academic and system-design concern regarding multi-agent AI coordination rather than an imminent national security threat.
• Autonomy: Full autonomy. The bots autonomously made decisions to edit and revert changes on Wikipedia pages without real-time human intervention.
• Novelty: Established threat. Simple algorithmic conflicts and feedback loops in multi-agent systems are well-documented and have occurred since the early days of the internet.

Impact by dimension

• Physical security: Negligible. The incident involves software bots editing Wikipedia articles, posing no physical threat to human safety, kinetic systems, or critical infrastructure.
• Information security: Negligible. While the incident involves information editing, it is limited to benign housekeeping bots on Wikipedia and does not represent state-sponsored disinformation or intelligence compromise.
• Sovereignty: Negligible. The bot interactions occurred on a public encyclopedia and had no impact on government operations, elections, or state sovereignty.
• Economic security: Negligible. The economic impact is negligible, limited to minor server resource inefficiencies for the Wikimedia Foundation, with no threat to strategic industries.
• Societal stability: Negligible. The incident caused minor frustration to some human editors but did not threaten societal stability, civil liberties, or human rights.