Nonconsensual Deepfake Porn of Bobbi Althoff Spreads Rapidly on X

Nonconsensual deepfake pornography of Bobbi Althoff, which had been in circulation for six months, is reported to have suddenly gone viral on X, jumping from around 178,000 views to 6.5 million views over a matter of hours. In addition to the harm to Althoff, this incident also spotlights X's role in distributing AI-generated nonconsensual porn due to alleged lax moderation.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The incident involved the creation of humiliating and sexually explicit synthetic imagery targeting a specific victim for engagement farming and personal gain, which aligns with the definition of targeted manipulation and harassment under sub-domain 4.3.

Additional risk subdomains

• 1.2 Exposure to toxic content: The viral spread of the graphic, non-consensual pornographic video exposed millions of social media users to highly inappropriate and toxic content.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The creation and distribution of the non-consensual deepfake video was an intentional act by human malicious actors using publicly available AI tools after their deployment.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes the creation and viral spread of an AI-generated deepfake video, which falls under the transparency obligations of Limited Risk systems under the EU AI Act.

AI system and alleged parties

• AI system: unspecified
• AI purpose: Deepfake Video Generation; Social Media Content Generation
• Behaviour type: Tool
• Alleged developer: Unnamed deepfake creators
• Alleged deployer: X (Twitter), Unnamed deepfake creators
• Alleged harmed parties: Bobbi Althoff

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Substantial, indirect Substantial
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes the creation and viral spread of a sexually explicit, non-consensual deepfake video.

Directly caused: A graphic, AI-generated video depicting Bobbi Althoff performing sexual acts was created and uploaded to X and specialized deepfake websites, accumulating over 6.5 million views on X.

Indirectly caused: The viral spread on X led to the video being reposted on other platforms like Reddit and drove a spike in traffic to specialized deepfake pornography websites.

Civil rights

Reported: The report explicitly describes how non-consensual deepfakes violate the consent and civil rights of women.

Directly caused: Bobbi Althoff's fundamental right to bodily autonomy, privacy, and consent was violated by the unauthorized digital recreation of her likeness in a sexual context.

Inferred additional harm: The widespread, unchecked proliferation of non-consensual deepfake technology disproportionately targets and silences women and girls, violating their civil rights to participate safely in online spaces.

Psychological

Reported: The report explicitly describes psychological distress, shock, and disgust experienced by the victim, Bobbi Althoff.

Directly caused: Bobbi Althoff experienced shock, disgust, and distress upon seeing the graphic video, stating that she had to 'cover her eyes' and was stunned that people believed it was real.

Indirectly caused: Althoff's PR team and family experienced concern and alarm, calling her to verify if the highly realistic video was real.

Inferred additional harm: Victims of non-consensual deepfakes often suffer long-term psychological trauma, intense depression, and anxiety due to the public violation of their consent and ongoing online harassment.

Epistemic

Reported: The report explicitly describes epistemic harm, as the highly realistic video misled users and the victim's team into believing it was a real 'leak'.

Directly caused: The high realism of the AI-generated video caused many social media users and Althoff's own team to initially believe the footage was a genuine leaked video.

Indirectly caused: The practice of labeling deepfakes as 'leaks' further manipulated public perception and eroded the ability of users to distinguish real media from synthetic media.

Inferred additional harm: The rapid advancement of indistinguishable deepfakes contributes to a broader erosion of shared truth, making visual evidence unreliable and fostering general skepticism toward real media.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 6500000

Potential causes

Management

• Gutting of Trust and Safety Teams: X management laid off moderation and curation staff, delaying responses.
• Prioritization of Chaos over Safety: Leadership publicly dismissed content moderation, encouraging online chaos.

Technology

• Advanced Face-Swapping AI: AI tools can seamlessly swap faces onto explicit videos with high realism.
• Accessibility of Deepfake Apps: No technical skills needed, just downloading an app or clicking buttons.

Data Inputs

• Stolen Original Video Footage: Using existing pornographic videos as base inputs for face-swapping.
• Public Likeness Data: Using public social media images and videos of victims to train AI.

Human Factors

• Financial Incentives for Users: Users share explicit content to farm engagement and monetize X accounts.
• Lack of User Critical Thinking: Users believed the realistic deepfake was a genuine leak and shared it.
• Malicious Intent of Creators: Creators share non-consensual porn to demean women and make money.

Process and Methods

• Slow Content Moderation on X: Platform took over 24 hours to remove viral non-consensual explicit videos.
• Ineffective Community Notes: Crowdsourced notes appear too late and do not stop the viral spread.
• Monetization of Viral Engagement: Platform ad-share payouts incentivize the spread of controversial media.

Regulatory Environment

• Lack of Federal Legislation: No US federal laws exist to criminalize making or sharing deepfakes.
• Absence of Platform Accountability: Platforms face no legal penalties for hosting non-consensual deepfakes.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the incident, including direct quotes from the victim, researchers, and platform policies. The facts regarding the creation, spread, and impact of the deepfake are clear and uncontested.
• Ambiguities identified: The specific AI application or model used to generate the deepfake is not identified in the reports.
• Alternative interpretations: None. The incident is clearly a case of non-consensual deepfake pornography used for harassment and engagement farming.

A viral non-consensual deepfake of a podcast host highlights the growing ease of generating highly realistic synthetic media and the challenges of platform content moderation. While causing severe individual psychological harm and violating personal rights, the incident has negligible direct national security implications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing societal and technological challenge rather than an imminent national security crisis.
• Autonomy: Human-controlled. The AI system was used as a passive tool by human creators to perform face-swapping under direct human control.
• Novelty: Established threat. Non-consensual celebrity deepfakes are an established threat with numerous prior occurrences.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety threats were involved in this social media incident.
• Information security: Minor. Demonstrates the scale and speed of viral synthetic media distribution on major platforms, though it lacks state-sponsored information warfare or intelligence compromise elements.
• Sovereignty: Negligible. No impact on state authority, government decision-making, or constitutional processes.
• Economic security: Negligible. No impact on strategic industries, financial systems, or national technological security.
• Societal stability: Minor. Violates individual human rights to privacy and consent, highlighting systemic issues with non-consensual deepfakes, but does not threaten large-scale societal stability.