San Francisco City Attorney Sues Operators of AI Deepfake Pornography Websites for Violations of State and Federal Law

In August 2024, the San Francisco City Attorney’s Office filed a first-of-its-kind lawsuit against 16 websites accused of using generative AI to create and distribute nonconsensual pornographic deepfakes, including images of minors. The sites, which had over 200 million visits in six months, invited users to upload real images to generate explicit fakes. By March 2025, 11 of the targeted websites had shut down following legal pressure.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The AI systems were used by malicious actors to create humiliating and sexual deepfake imagery of specific victims for harassment, bullying, and extortion.

Additional risk subdomains

• 1.2 Exposure to toxic content: The websites facilitated the generation and exposure of users to highly toxic, abusive, and illegal content, including nonconsensual pornography and child sexual abuse material.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The harm was caused by bad actors intentionally training and using deployed generative AI models to create and distribute nonconsensual pornography.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: Limited Risk. The report describes 'AI-generated nonconsensual explicit images' and 'deepfake pornographic images', which fall under the category of 'AI-generated content such as deepfakes' requiring transparency obligations under the EU AI Act.

AI system and alleged parties

• AI system: unspecified
• AI purpose: Deepfake Image Generation; Image Generation
• Behaviour type: Tool
• Alleged developer: Unknown deepfake technology developers
• Alleged deployer: Operators of 16 deepfake pornography websites
• Alleged harmed parties: Women depicted in deepfake pornography, Minors depicted in deepfake pornography

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Substantial, indirect Substantial
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Substantial, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Substantial, indirect Negligible
• Psychological: direct Substantial, indirect Negligible
• Epistemic: direct Substantial, indirect Negligible
• Child sexual exploitation and abuse: direct Severe, indirect Substantial

Malicious content

Reported: The report explicitly describes the creation and spread of toxic and malicious content.

Directly caused: The websites facilitated the generation of nonconsensual explicit pornographic images and child sexual abuse material.

Indirectly caused: AI-generated nonconsensual nude images of 16 eighth-grade students were circulated among students at a California middle school.

Inferred additional harm: Given the 200 million visits to these websites, millions of toxic, sexually explicit deepfake images were likely generated and distributed online.

Differential treatment

Reported: The report explicitly describes how this phenomenon disproportionately targets and exploits women and girls.

Directly caused: The lawsuit alleges that the proliferation of nonconsensual deepfake pornographic images has specifically exploited real women and girls.

Inferred additional harm: The widespread availability of these tools reinforces gender-based harassment and unequal safety outcomes for women and girls in digital spaces.

Civil rights

Reported: The report describes violations of state and federal laws prohibiting deepfake pornography, revenge pornography, and child pornography.

Directly caused: The generation of nonconsensual pornography and child sexual abuse material violates the victims' fundamental rights to privacy and bodily autonomy.

Inferred additional harm: The systemic exploitation of women and children through these tools represents a widespread violation of basic human rights and digital safety.

Privacy

Reported: The report explicitly describes privacy violations resulting from the incident.

Directly caused: The websites allowed users to upload clothed photos of real, identifiable individuals to generate nude versions without their consent.

Inferred additional harm: Thousands of individuals had their personal likenesses compromised and sexualized without their knowledge or consent, leading to permanent violations of their digital privacy.

Psychological

Reported: The report explicitly describes psychological harm to victims of nonconsensual deepfake pornography.

Directly caused: The report states that the generated images are used to extort, bully, threaten, and humiliate women and girls, causing immeasurable harm.

Inferred additional harm: It is highly likely that thousands of victims whose images were generated suffered severe emotional distress, anxiety, depression, and trauma.

Epistemic

Reported: The report notes that the generated images are virtually indistinguishable from real photographs.

Directly caused: The creation of highly realistic, fake pornographic images of real people undermines the ability to distinguish real media from synthetic fabrications.

Inferred additional harm: The proliferation of indistinguishable deepfakes contributes to a broader erosion of trust in digital media and photographic evidence.

Child sexual exploitation and abuse

Reported: The report explicitly describes CSEA incidents involving the targeted websites.

Directly caused: The lawsuit alleges that some of the websites allowed users to create nonconsensual pornographic images of children, and that bad actors trained models on child sexual abuse materials.

Indirectly caused: AI-generated nonconsensual nude images of 16 eighth-grade students were circulated among students at a California middle school.

Inferred additional harm: It is highly likely that numerous other minors were subjected to AI-generated sexual exploitation and abuse through these 'undress' websites.

People affected

• Occurrences reported: 1
• People reportedly harmed: 18
• People reportedly exposed: 18

Potential causes

Management

• Prioritizing Profit Over Safety: Website operators design platforms specifically to exploit and monetize targets.

Technology

• Open-Source Generative AI Models: Free, sophisticated models are easily exploited and retrained by bad actors.
• User-Friendly 'Undress' Interfaces: Websites lower technical barriers to generate nonconsensual explicit images.
• Lack of Watermarking or Tracking: No digital markings on generated photos make it impossible to trace creators.

Data Inputs

• Pornographic Training Datasets: Models are further trained on explicit images and child sexual abuse material.
• Unconsenting Clothed Photos: Users upload real, clothed images of targets without their consent.

Human Factors

• Malicious Intent of Users: Bad actors use images to extort, bully, threaten, and humiliate victims.
• Profit Motivation of Operators: Website owners charge fees to monetize nonconsensual pornography.

Process and Methods

• Inadequate Website Verification: No processes to verify consent or age of subjects in uploaded photos.
• Lack of Content Moderation: Platforms fail to block generation of explicit content involving minors.

Regulatory Environment

• Delayed Legal Action: Slow enforcement allowed websites to amass hundreds of millions of views.
• Jurisdictional Challenges: Anonymity and decentralized hosting make it difficult to prosecute creators.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, detailed factual accounts of the lawsuit, the specific websites targeted, the mechanism of the AI's misuse (undressing clothed photos), and specific examples of victims (eighth-grade students, Taylor Swift). The legal outcomes (shutdown of 11 websites) are also clearly documented.
• Ambiguities identified: The specific open-source AI models used and the exact technical methods of training them on CSAM are not detailed.
• Alternative interpretations: None. The incident is clearly a case of malicious misuse of generative AI for nonconsensual pornography.

The incident involves the malicious use of generative AI to create nonconsensual pornographic deepfakes targeting adults and minors. While representing a severe societal issue and law enforcement challenge, its direct impact on core national security functions remains minor and manageable under standard legal and regulatory frameworks.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing, strategic regulatory and law enforcement challenge regarding generative AI capabilities rather than an active, imminent national security crisis.
• Autonomy: Human-controlled. The AI systems function as tools that require human users to actively upload photos and initiate the generation process.
• Novelty: Evolved capability. Represents a significant scaling and refinement of existing deepfake and 'undress' technologies, leveraging open-source models trained on illicit datasets.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic targeting capabilities were impacted or threatened in this incident.
• Information security: Negligible. While involving deepfakes, the incident represents localized criminal harassment and commercial exploitation rather than a coordinated state-sponsored information warfare or intelligence operation.
• Sovereignty: Negligible. No threat to state sovereignty, electoral integrity, or core government decision-making processes. The incident was addressed through standard state and federal legal channels.
• Economic security: Negligible. No significant threat to national financial systems, strategic technologies, or core economic infrastructure was reported.
• Societal stability: Minor. The incident represents a minor national security concern but a significant societal issue, involving widespread privacy violations, harassment of women and minors, and the generation of illegal materials.