Facebook "News Feed" Allegedly Boosted Misinformation and Violating Content Following Use of MSI Metric

After the “News Feed” algorithm had been overhauled to boost engagement between friends and family in early 2018, its heavy weighting of re-shared content was alleged found by company researchers to have pushed content creators to reorient their posts towards outrage and sensationalism, causing a proliferation of misinformation, toxicity, and violent content.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.2 Pollution of information ecosystem and loss of consensus reality

The algorithm overhaul caused ecosystem-scale degradation of public discourse, forcing political parties and publishers to adopt inflammatory tactics and polluting the information ecosystem with toxic, divisive content.

Additional risk subdomains

• 1.2 Exposure to toxic content: The algorithm directly amplified and exposed users to a higher prevalence of insults, threats, and angry, divisive content.
• 7.3 Lack of capability or robustness: The algorithm failed to perform safely under real-world conditions, as its optimization metrics were easily exploited and gamed by political actors and publishers.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The harms were caused by the ranking decisions of the News Feed algorithm, which produced unintended negative side effects (amplifying outrage) while pursuing the goal of increasing meaningful social interactions.

EU AI Act risk tier

• Risk tier: 1 Unacceptable

Risk Level 1: Unacceptable Risk. The report describes an AI system used for behavioral influence and optimization that led to significant societal harm, including political polarization, the spread of toxic content, and the facilitation of civil unrest.

AI system and alleged parties

• AI system: News Feed algorithm (Meta)
• AI purpose: Content Recommendation; Behavioral Modeling
• Behaviour type: Autonomous
• Alleged developer: Facebook
• Alleged deployer: Facebook
• Alleged harmed parties: Facebook users, Facebook content creators

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Substantial
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Substantial
• Financial: direct Negligible, indirect Substantial
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Substantial, indirect Severe
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Substantial
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Severe
• Epistemic: direct Severe, indirect Severe
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Physical

Reported: The report explicitly describes physical harm indirectly caused by the incident through the mention of the January 6 Capitol riot.

Indirectly caused: The report notes that election protesters used the platform to organize protests that led to the January 6 riot at the Capitol, which historically resulted in multiple injuries and fatalities.

Inferred additional harm: It is highly likely that the amplification of divisive and violent content contributed to physical violence and civil unrest in other highly sensitive regions mentioned, such as Ethiopia and Myanmar.

Property

Reported: The report explicitly describes property damage indirectly caused by the incident through the mention of the January 6 Capitol riot.

Indirectly caused: The January 6 Capitol riot, which was organized using the platform, resulted in physical damage to the U.S. Capitol building.

Financial

Reported: The report explicitly describes financial losses for online publishers due to massive traffic declines caused by the algorithm change.

Indirectly caused: Online publishers suffered severe traffic drops, with Topix losing 53% of its traffic (leading to its acquisition), Breitbart losing 46%, BuzzFeed losing 13%, and ABC News losing 12%, resulting in significant lost advertising revenue.

Inferred additional harm: The sudden algorithmic shift likely caused millions of dollars in lost advertising and subscription revenue for publishers globally who relied on Facebook for distribution.

Malicious content

Reported: The report explicitly describes the spread of toxic, violent, and malicious content directly and indirectly caused by the algorithm.

Directly caused: The algorithm directly prioritized and amplified toxic content, reshares containing misinformation, and insults, resulting in a 43% increase in insults and threats on Spanish public pages.

Indirectly caused: The algorithm's incentives forced publishers and political parties to produce more negative, sensationalist, and inflammatory content to maintain their reach.

Inferred additional harm: Massive global proliferation of toxic, hateful, and divisive content across Facebook's user base of nearly three billion people.

Democracy

Reported: The report explicitly describes the erosion of democratic norms caused indirectly by the algorithm's incentives.

Indirectly caused: The algorithm incentivized political parties in Poland, Spain, Taiwan, and India to shift toward negative campaigning, direct attacks, and inflammatory posts, undermining democratic discourse and facilitating the January 6 Capitol riot.

Inferred additional harm: Systemic, long-term erosion of democratic stability, public trust in institutions, and social cohesion across multiple democratic nations globally.

Psychological

Reported: The report explicitly mentions concerns regarding mental health and the platform becoming an 'angrier place' with unhealthy side effects.

Indirectly caused: The algorithm's promotion of outrage, insults, and threats made the platform significantly more toxic, likely causing emotional distress, anger, and anxiety for millions of users.

Inferred additional harm: Widespread psychological distress and negative mental health impacts are inferred across a significant portion of Facebook's nearly three billion users due to the systematic amplification of hostile and divisive content.

Epistemic

Reported: The report explicitly describes epistemic harm, noting that misinformation and lies were inordinately prevalent and rewarded by the algorithm.

Directly caused: The algorithm directly boosted false content, 'fad/junky science', and misinformation because they generated high levels of comments and reshares.

Indirectly caused: Political operatives and publishers relied more on sensationalism and lies to exploit the algorithm's ranking metrics for distribution.

Inferred additional harm: Widespread degradation of the shared information ecosystem, leading to millions of users believing false civic, political, and health information.

People affected

• Occurrences reported: 1
• People reportedly exposed: 3000000000

Potential causes

Management

• Prioritizing Growth Over Safety: Management resisted fixes that would cause material tradeoffs with MSI.
• Fear of Declining Engagement: Leadership feared users would leave the app if engagement continued to fall.
• Resistance to Integrity Proposals: Executives blocked broad deployment of deep reshare mitigation.

Technology

• Flawed MSI Ranking Formula: The algorithm heavily weighted comments and reshares, promoting outrage.
• Deep Reshare Amplification: Algorithm boosted content shared across long chains without verification.
• Downstream MSI Optimization: Optimizing for downstream sharing rewarded highly controversial posts.

Data Inputs

• Engagement-Based Metrics: Using raw shares and comments as proxies for meaningful interaction.
• Lack of Sensitive Content Signals: Algorithm lacked inputs to distinguish healthy debate from toxic outrage.
• Absence of Civic Content Signals: Initial lack of specialized signals for sensitive categories like politics.

Human Factors

• Outrage Bias in User Engagement: Users are naturally more likely to comment on and share divisive content.
• Publisher Adaptation to Outrage: Publishers shifted to sensationalism to maintain distribution.
• Political Party Exploitation: Political actors used negative campaigning to maximize platform reach.

Process and Methods

• Inadequate Impact Testing: Facebook deployed major algorithm changes without assessing societal risks.
• Delayed Mitigation Deployment: Integrity team fixes were delayed or restricted to preserve metrics.
• Overreliance on Engagement Surveys: Surveys focused on business health over systemic platform toxicity.

Regulatory Environment

• Absence of Algorithmic Regulation: No regulatory framework governed the design of recommendation algorithms.
• Lack of External Oversight: No independent auditing of platform engagement metrics and impacts.

Information quality

• Classification confidence: High
• Reason for confidence: The assessment is based on a highly detailed investigative report by The Wall Street Journal, which reviewed internal Facebook memos, data science studies, and executive communications. This provides direct, authoritative evidence of the algorithm's design, its unintended consequences, and the internal awareness of these issues. There is very little ambiguity regarding the system's role or the resulting harms.
• Ambiguities identified: The exact financial losses of the affected publishers are not fully quantified in dollar terms, only in percentage traffic drops.
• Alternative interpretations: Facebook executives argued that societal divisions were growing long before social media, suggesting the algorithm was not the primary cause of polarization but merely reflected existing societal trends.

Facebook's 2018 News Feed algorithm overhaul unintentionally prioritized sensationalist, toxic, and misinformative content to drive engagement. This systemic amplification polluted the global information ecosystem, altered political campaigning strategies in multiple democracies, and facilitated the organization of the January 6 Capitol riot, posing notable threats to democratic sovereignty and societal stability.

• Overall national security impact: Substantial
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: United States, Spain, Poland, India, Taiwan
• Alleged perpetrator: Facebook

Threat characteristics

• Imminence: Long-term. The incident represents a long-term strategic concern regarding algorithmic recommendation systems and their ongoing impact on democratic discourse, rather than an active 72-hour crisis.
• Autonomy: Full autonomy. The recommendation algorithm operated fully autonomously to rank and curate personalized content for billions of users in real-time without human intervention in individual decisions.
• Novelty: Evolved capability. While recommendation algorithms were established, optimizing for meaningful social interactions represented an evolved capability that significantly amplified negative societal feedback loops.

Impact by dimension

• Physical security: Minor. The algorithm indirectly facilitated the organization of protests that led to physical violence and property damage during the January 6 Capitol riot, though it did not directly control kinetic systems.
• Information security: Substantial. The algorithm systematically prioritized and amplified sensationalist, toxic, and misinformative content, significantly polluting the information ecosystem and enabling widespread automated information manipulation.
• Sovereignty: Substantial. The algorithmic incentives disrupted democratic processes by forcing political parties in multiple nations to adopt negative campaigning tactics and indirectly contributing to the disruption of the US constitutional process on January 6.
• Economic security: Minor. Algorithmic shifts caused significant traffic and revenue losses for global publishers, though it did not pose a direct threat to critical financial infrastructure or strategic industries.
• Societal stability: Substantial. The platform became an angrier place, systematically amplifying outrage, insults, and threats, which eroded social cohesion and manipulated societal stability at a population scale.