Instagram's Algorithm Reportedly Recommended Sexual Content to Teenagers' Accounts

Tests by The Wall Street Journal and a researcher reportedly revealed that Instagram's AI-driven Reels algorithm recommends sexually suggestive content to accounts listed as 13 years old. Despite Meta's commitment to restrict such content for minors, explicit videos were served within minutes of account creation, according to the findings.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The Instagram Reels recommendation algorithm exposed minor users to highly inappropriate, sexually suggestive, and explicit content, violating platform guidelines and community norms.

Additional risk subdomains

• 1.3 Unequal performance across groups: Internal data showed teens were disproportionately exposed to nudity, violence, and bullying compared to adult users over 30.
• 7.3 Lack of capability or robustness: Meta's automated content filtering and age-gating recommendation systems failed to reliably perform under testing conditions.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The Reels recommendation algorithm post-deployment made decisions to serve sexually explicit content to teen accounts to maximize engagement, which was an unintentional outcome of its optimization goals.

EU AI Act risk tier

• Risk tier: 1 Unacceptable

Unacceptable Risk: The report describes an AI system that exploits the vulnerabilities of minor users based on their age by autonomously serving them highly inappropriate, sexually explicit content, which poses significant psychological risks.

AI system and alleged parties

• AI system: Instagram Reels algorithm (Meta)
• AI purpose: Content Recommendation; Content Curation
• Behaviour type: Autonomous
• Alleged developer: Meta Platforms, Instagram
• Alleged deployer: Meta Platforms, Instagram
• Alleged harmed parties: minors, Instagram users, General public, Children

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Substantial, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Substantial
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Minor

Malicious content

Reported: The report explicitly describes the Reels algorithm recommending sexually suggestive and explicit content, nudity, violence, and bullying to minors.

Directly caused: The Reels algorithm directly served and promoted sexually explicit videos, promotions for adult sex-content creators, and videos depicting violence and bullying to test accounts.

Inferred additional harm: It is inferred that vast amounts of toxic, sexually explicit, and violent content are continuously served to millions of underage users worldwide due to these algorithmic recommendation failures.

Differential treatment

Reported: The report explicitly describes systematic differential treatment of teenagers compared to adults regarding exposure to harmful content.

Directly caused: The Reels algorithm served teens three times as many prohibited posts containing nudity, 1.7 times as much violence, and 4.1 times as much bullying content as users above the age of 30.

Inferred additional harm: This differential exposure results in systemic unequal safety outcomes for minor users compared to adult users on the platform.

Psychological

Reported: The report mentions that teens on Instagram reported exposure to bullying, violence, and unwanted nudity at rates exceeding older users, which can cause psychological distress, but does not explicitly quantify the psychological harm.

Indirectly caused: The exposure of minors to highly explicit sexual content, bullying, and violence can lead to psychological distress, anxiety, and developmental harm, though specific cases are not detailed.

Inferred additional harm: It is highly likely that a significant portion of the millions of teen users exposed to this content experienced psychological distress, anxiety, or trauma, estimated to affect hundreds of thousands of minors globally.

Child sexual exploitation and abuse

Reported: The report does not explicitly describe a CSEA incident.

Inferred additional harm: The recommendation of adult content creators who offer direct messages and nude photos to 13-year-old accounts creates a high-risk environment facilitating potential grooming or child sexual exploitation, though no specific cases are detailed in the report.

People affected

• Occurrences reported: 1

Potential causes

Management

• Tight Safety Budgets: Years of tight budgets limited investments in robust child-safety work.
• Rejection of System Separation: Management did not pursue building a separate recommendation system for teens.
• Insufficent Policy Enforcement: Failed to enforce announced policy of blocking sensitive content for minors.

Technology

• Engagement-Biased Algorithm: Reels recommended edgier content based on video completion and dwell time.
• Weak Automated Filtering: Automated systems were too weak to filter prohibited content for teens.
• Shared Recommendation Pipeline: Lacked a separate recommendation system designed specifically for teen safety.

Data Inputs

• System Classification Errors: Recommended videos already labeled as disturbing to underage accounts.
• Lack of Content Differentiation: System failed to properly filter out sensitive inputs for teenage feeds.

Human Factors

• User Engagement Triggers: Normal teen browsing behavior like lingering on videos triggered explicit feeds.

Process and Methods

• Inadequate Default Restrictiveness: Default settings failed to block sexually suggestive content effectively.
• Insufficient Testing Protocols: Internal tests identified issues years prior but fixes were not fully realized.

Regulatory Environment

• Underappreciated Legal Risks: Meta attorneys noted child-safety posed unappreciated regulatory risks.

Information quality

• Classification confidence: High
• Reason for confidence: The report is highly detailed, based on rigorous independent testing by the Wall Street Journal and a Northeastern University professor, as well as leaked internal Meta documents confirming the findings. The facts are clear, consistent, and uncontested by technical evidence, with Meta only disputing the representativeness of the simulation.
• Ambiguities identified: The exact number of real-world teenagers exposed or harmed is not quantified, and the precise algorithmic weights that led to the rapid escalation of explicit content are not fully detailed.
• Alternative interpretations: Meta argued that the tests were artificial simulations that do not reflect real-world teen behavior, suggesting the system might perform differently under typical usage.

Instagram's autonomous Reels recommendation algorithm was found to systematically serve sexually explicit and harmful content to minor accounts. While presenting significant child safety and corporate governance concerns, the incident has negligible direct national security implications, representing a minor societal safety issue manageable through regulatory frameworks.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing systemic platform safety and regulatory concern rather than an immediate national security crisis.
• Autonomy: Full autonomy. The Reels recommendation algorithm operates fully autonomously to curate and serve content based on user interaction metrics without human intervention.
• Novelty: Established threat. Algorithmic recommendation failures and child safety challenges on major social media platforms are well-documented, established issues.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic targeting capabilities were affected in this incident.
• Information security: Negligible. No evidence of foreign disinformation campaigns, intelligence compromises, or classified data theft associated with this algorithmic failure.
• Sovereignty: Negligible. No disruption to state authority, electoral systems, or core government decision-making processes.
• Economic security: Negligible. No direct threats to financial systems, strategic technologies, or critical economic supply chains.
• Societal stability: Minor. Systemic exposure of minor users to explicit and harmful content represents a widespread population safety concern, though it remains manageable through regulatory and platform enforcement.