Offensive Instagram User Content Displayed as Facebook Ad

An Instagram user’s image containing violent content was reportedly used as advertisement on Facebook allegedly via automated means.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The AI system exposed users (the victim's sister and friends) to highly toxic, abusive, and violent content (a rape and death threat) by automatically generating an advertisement from the post.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The algorithm lacked the capability to analyze text within images and filter out violent or abusive content before using it for promotions.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by Instagram's automated engagement and advertising algorithms selecting and promoting an abusive post post-deployment without intending to spread toxic content.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The system is an automated ad delivery and engagement algorithm, which generally poses low or negligible risk to society under the EU AI Act.

AI system and alleged parties

• AI system: Instagram engagement algorithm (Meta)
• AI purpose: Ad Delivery; Content Recommendation
• Behaviour type: Autonomous
• Alleged developer: Facebook
• Alleged deployer: Facebook
• Alleged harmed parties: Olivia Solon's Facebook connections, Olivia Solon

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the report describes the automated promotion of a post containing a violent rape threat.

Directly caused: Instagram's algorithm generated an advertisement featuring the text 'I will rape you before I kill you, you filthy whore!' and distributed it to the victim's sister and friends.

Inferred additional harm: The automated system likely served similar highly engaging but toxic posts to other users' networks, though not detailed in this specific report.

Psychological

Reported: Yes, the report describes the victim and her family being distressed by the automated promotion of a violent rape threat.

Directly caused: Olivia Solon expressed distress and sarcasm regarding Facebook reminding her and her family of the rape threat during her honeymoon.

Inferred additional harm: Her sister and other friends who saw the ad likely experienced distress or discomfort upon seeing a violent threat targeted at Solon.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 2

Potential causes

Management

• Prioritization of Growth Over Safety: Management prioritized platform engagement over robust content filtering.
• Inadequate Risk Assessment: The risks of automated content promotion were not properly evaluated.

Technology

• Engagement-driven Ad Algorithm: The algorithm promoted posts solely based on high user engagement metrics.
• Lack of Image Text Filtering: Systems failed to analyze and filter violent text embedded within images.

Data Inputs

• Misinterpreted Engagement Signals: Likes and comments on a threat post were misconstrued as positive engagement.
• No OCR Metadata Integration: Text within the screenshot was not ingested as metadata for safety filtering.

Human Factors

• System Over-reliance: Operators relied fully on automation to select promotional content.

Process and Methods

• No Human-in-the-Loop Review: Promotional notifications were dispatched without manual safety checks.
• Inadequate Ad Safety Policies: Processes lacked specific guardrails for automated engagement ads.

Regulatory Environment

• Lack of Algorithmic Oversight: No regulatory standards existed to govern automated ad-targeting safety.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, firsthand accounts from the affected journalist and official statements from Instagram/Facebook confirming the mechanics of the incident.
• Ambiguities identified: It is slightly unclear how many of Solon's friends actually saw the ad, as Instagram only specified it went to a 'small percentage'.

An automated Instagram algorithm mistakenly promoted a post containing a violent rape threat to a user's friends due to high engagement metrics. This incident represents a corporate content moderation failure with negligible national security implications, as it did not involve state actors, critical infrastructure, or population-scale manipulation.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents a past, localized corporate product failure rather than an active, developing national security threat.
• Autonomy: Full autonomy. The advertising system acted fully autonomously to select, format, and distribute the post without human-in-the-loop oversight.
• Novelty: Established threat. Algorithmic amplification of high-engagement but inappropriate content is an established threat and known failure mode of social media recommendation systems.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety elements were impacted or threatened by this automated advertising incident.
• Information security: Negligible. No foreign disinformation campaign, intelligence compromise, or state-sponsored information operations were involved in this localized algorithmic failure.
• Sovereignty: Negligible. The incident was a corporate algorithmic error and did not impact state authority, electoral processes, or government decision-making.
• Economic security: Negligible. No strategic technology theft, critical supply chain disruption, or financial system attacks occurred as a result of this incident.
• Societal stability: Negligible. While the incident caused distress to an individual and her network, it does not represent population-scale social manipulation or systemic civil rights violations.