Instagram Algorithms Allegedly Promote Accounts Facilitating Child Sex Abuse Content

An investigation disclosed that Instagram's recommendation algorithms are promoting accounts that facilitate and sell child sexual abuse material (CSAM). The study, conducted by The Wall Street Journal and researchers at Stanford University and the University of Massachusetts Amherst, indicates that Instagram's algorithms not only allow for the discovery of such accounts through keyword searches but also actively recommend them to users within the network. The issue is especially concerning given Instagram's popularity among teenagers.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The Instagram recommendation algorithm actively promoted accounts facilitating child sexual abuse material, exposing users to highly toxic, illegal, and harmful content.

Additional risk subdomains

• 7.3 Lack of capability or robustness: Twitter's automated scanning for known CSAM broke and failed to perform reliably, representing a severe lack of capability and robustness in content moderation.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the post-deployment behavior of Instagram's recommendation algorithm and the failure of Twitter's automated scanning system, which produced unintended harmful outcomes.

EU AI Act risk tier

• Risk tier: 1 Unacceptable

Unacceptable Risk: The report describes exploitative systems targeting vulnerabilities based on age (minors) and facilitating child sexual exploitation, which violates fundamental rights.

AI system and alleged parties

• AI system: Instagram recommendation algorithms, Twitter basic scanning (Meta)
• AI purpose: Content Recommendation; Content Moderation
• Behaviour type: Autonomous
• Alleged developer: Meta, Instagram
• Alleged deployer: Meta, Instagram
• Alleged harmed parties: teenagers, minors, General public, Children

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Substantial, indirect Substantial
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Substantial
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Severe, indirect Severe

Malicious content

Reported: Yes, the report explicitly describes the promotion and spread of child sexual abuse material (CSAM) on both platforms.

Directly caused: Instagram's recommendation algorithms actively promoted accounts facilitating and selling self-generated child sexual abuse material (SG-CSAM).

Indirectly caused: Twitter's failure to scan for known CSAM allowed such toxic content to remain available on the platform for months.

Inferred additional harm: It is inferred that a massive volume of toxic and illegal CSAM was distributed to an untracked number of users across both platforms due to these systemic failures.

Civil rights

Reported: Yes, the exploitation of children and the distribution of CSAM represents a severe violation of fundamental human and civil rights.

Indirectly caused: The promotion and facilitation of CSAM networks directly facilitated the violation of the fundamental rights of the children depicted and exploited.

Inferred additional harm: Thousands of children likely suffered severe violations of their fundamental rights to safety, privacy, and protection from exploitation due to these platform failures.

Child sexual exploitation and abuse

Reported: Yes, the report explicitly describes a major CSEA incident involving the promotion and facilitation of CSAM and SG-CSAM.

Directly caused: Instagram's recommendation algorithm promoted accounts selling SG-CSAM, and Twitter failed to scan for known CSAM.

Indirectly caused: These algorithmic failures facilitated the discovery and transaction of CSAM by buyers and sellers.

Inferred additional harm: It is highly likely that these platform failures facilitated the ongoing abuse, exploitation, and distribution of material depicting thousands of children.

People affected

• Occurrences reported: 2

Potential causes

Management

• Delayed Safety Action: Meta set up an internal task force only after research was published.
• Failure to Fulfill Pledges: Twitter failed to address CSAM despite executive pledges to do so.
• Blocking External Researchers: Management cut off API access for researchers studying CSAM.

Technology

• Recommendation Algorithm Promotion: Algorithms promoted SG-CSAM accounts to users viewing network accounts.
• Broken CSAM Scanning: Twitter's basic scanning for known CSAM broke after the takeover.
• Widespread Hashtag Use: Widespread use of hashtags facilitated discovery of explicit accounts.

Data Inputs

• Self-Generated CSAM Content: Accounts uploaded SG-CSAM pretending to be operated by minors.
• Explicit Hashtags: Explicit hashtags were used to label and find illicit accounts.
• Restricted API Data Access: Twitter cut off API access, blocking research into platform data.

Human Factors

• Active Buyer and Seller Network: Malicious actors actively used platforms to buy and sell CSAM.
• High Teen Platform Popularity: Instagram's popularity among teens made it a target ecosystem.

Process and Methods

• Algorithmic Account Suggestion: Recommendations allowed discovery without requiring keyword searches.
• Inadequate Content Review: Accounts offering CSAM remained active for months without removal.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly details the algorithmic failures of both Instagram and Twitter regarding CSAM promotion and scanning, backed by a Stanford University study. The roles of the AI systems are explicitly stated, leaving little ambiguity about the nature of the failures.
• Ambiguities identified: The exact number of affected minors and the volume of CSAM transactions facilitated are not quantified in the text.

Failures in Instagram's autonomous recommendation algorithms and Twitter's broken automated scanning systems facilitated the promotion and discovery of child sexual abuse material networks. While representing a severe human rights violation and a significant regulatory challenge, the national security impact remains minor and manageable within standard law enforcement and legal frameworks.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic and regulatory concern regarding platform safety rather than an immediate national security crisis.
• Autonomy: Full autonomy. The recommendation algorithms operated autonomously to connect and promote accounts without direct human oversight or intervention.
• Novelty: Evolved capability. Represents a significant and concerning evolution of algorithmic recommendation failures leading to the automated promotion of illicit networks.

Impact by dimension

• Physical security: Negligible. The incident involves algorithmic failures on social media platforms and does not pose any threat to physical systems, critical infrastructure, or human safety through kinetic means.
• Information security: Negligible. No evidence of state-sponsored information warfare, intelligence compromise, or systematic disinformation campaigns targeting national institutions.
• Sovereignty: Negligible. The platform failures do not impact state authority, territorial control, electoral systems, or core government decision-making processes.
• Economic security: Negligible. No significant threat to financial systems, strategic technology theft, or national economic stability was reported.
• Societal stability: Minor. While representing severe human rights violations regarding child exploitation, the issue is primarily a law enforcement and regulatory matter rather than a direct threat to national societal stability.