Lensa AI's Produced Unintended Sexually Explicit or Suggestive "Magic Avatars" for Women

Lensa AI's "Magic Avatars" were reportedly generating sexually explicit and sexualized features disproportionately for women and Asian women despite not submitting any sexual content.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The Lensa AI system directly generated and exposed users to highly inappropriate, sexually explicit, and toxic content, including unsolicited nudes of themselves and sexualized images of children.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The app systematically altered users' features based on gender and race, such as lightening skin, anglicizing features, and applying sexualized stereotypes to women.
• 6.3 Economic and cultural devaluation of human effort: The underlying Stable Diffusion model scraped and reproduced copyrighted artistic styles without consent, devaluing the work of digital artists.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The generation of sexually explicit, biased, and child-sexualized images was an unexpected and unintended outcome of deploying the Lensa AI avatar feature.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The app generates synthetic images and deepfakes of users, which under the EU AI Act are subject to transparency obligations to ensure users are aware the content is AI-generated.

AI system and alleged parties

• AI system: Lensa AI, Stable Diffusion (Prisma Labs, Stability AI)
• AI purpose: Image Generation; Deepfake Image Generation
• Behaviour type: Tool
• Alleged developer: Stability AI, Runway, Lensa AI, LAION, EleutherAI, CompVis LMU
• Alleged deployer: Lensa AI
• Alleged harmed parties: women using Lensa AI, Asian women using Lensa AI

Harm severity

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Minor, indirect Negligible

Malicious content

Reported: The report explicitly describes the creation of toxic and sexually explicit content by the AI system.

Directly caused: The AI directly generated and spread (via user downloads) sexually explicit images, including nude avatars of adults and children, and images with 'ahegao' faces.

Indirectly caused: N/A

Inferred additional harm: It is likely that millions of sexualized or inappropriate images were generated and potentially shared across social media platforms.

Differential treatment

Reported: The report explicitly describes systematic differential treatment based on gender and race.

Directly caused: Female avatars were systematically sexualized (given cosmic G-strings, larger breasts) compared to male avatars (who got space suits and professional attire). Black and Asian users had their skin lightened or features anglicized.

Indirectly caused: N/A

Inferred additional harm: Widespread reinforcement of racial and gender stereotypes among millions of users who interacted with the app.

Civil rights

Reported: The report explicitly describes concerns regarding non-consensual sexualization and potential for revenge porn.

Directly caused: Generation of sexually explicit images of individuals without their explicit consent for such content (even if they uploaded selfies).

Indirectly caused: N/A

Inferred additional harm: Potential violations of bodily autonomy and privacy rights for users whose likenesses were sexualized.

Psychological

Reported: The report explicitly describes psychological distress and feelings of violation experienced by users.

Directly caused: Users, particularly women, reported feeling 'very violated' after receiving unwanted nude or highly sexualized avatars of themselves.

Indirectly caused: Users experienced disappointment and distress due to the app altering their bodies to be thinner or lighter-skinned, reinforcing negative body image standards.

Inferred additional harm: It is highly likely that thousands of female and minority users experienced distress, body image issues, or feelings of violation from the unsolicited sexualization and racial alterations of their likenesses.

Child sexual exploitation and abuse

Reported: The report explicitly describes a CSEA incident where a reporter tested the app with childhood photos and received sexualized, semi-nude images of herself as a child.

Directly caused: The app generated semi-nude and sexualized images of a child when childhood photos were uploaded, bypassing its own age filters.

Indirectly caused: N/A

Inferred additional harm: It is highly likely other users or malicious actors generated similar sexualized imagery of minors, creating deepfaked CSEM.

People affected

• Occurrences reported: 1
• People reportedly harmed: 15
• People reportedly exposed: 20000000

Potential causes

Management

• Prioritization of Rapid Growth: App launched without safety layers to capture market demand quickly.
• Failure to Enforce Terms: Prisma Labs did not actively block uploads of minors' or nude photos.
• Inadequate Risk Assessment: Management failed to foresee generation of deepfakes and child imagery.

Technology

• Unfiltered Stable Diffusion Model: Open-source model lacks technical constraints to block unsafe generations.
• Biased CLIP Neural Network: CLIP associates women with sexual content and men with professional roles.
• Disabled or Bypassed NSFW Filter: The app lacked active filters, allowing generation of explicit images.

Data Inputs

• Uncurated LAION-5B Dataset: Dataset contains billions of internet images with deep societal biases.
• Non-Consensual Artist Data: Training data used copyrighted art without permission or licensing.
• Lack of Input Photo Verification: System does not verify if uploaded photos belong to users or minors.

Human Factors

• User Violations of Terms: Users uploaded photos of minors and photoshopped images to bypass rules.
• Societal Biases in Internet Data: Human prejudices online were captured in data and mirrored by the AI.
• Intentional Harassment Motives: Bad actors exploit the tool to generate non-consensual revenge porn.

Process and Methods

• Inadequate Content Moderation: Lack of active output filtering allowed explicit images to reach users.
• Flawed Gender Classification: Binary gender selection triggers pre-defined sexualized artistic styles.
• Absence of Artist Opt-Out Process: No mechanism was provided for artists to remove their work from training.

Regulatory Environment

• Lack of AI Content Regulation: No legal frameworks govern the generation of AI-synthesized pornography.
• Unregulated Data Scraping: Weak copyright laws allowed scraping of artists' work without consent.
• No Age Verification Mandates: Absence of strict legal requirements to verify user age before generation.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed, and first-hand accounts of Lensa AI's behavior, including specific tests run by journalists that confirmed the generation of explicit content and biased outputs. The technical mechanism is well-documented, and Prisma Labs' responses are included.
• Ambiguities identified: The exact proportion of users who received explicit or biased images is not quantified in the reports.
• Alternative interpretations: None. The evidence of bias and unwanted sexualization is clear and replicated across multiple independent reports.

Lensa AI's Magic Avatars feature demonstrated significant gender and racial biases, alongside generating non-consensual explicit imagery and child-related sexualized content. While representing notable societal and privacy concerns, the incident has negligible direct impact on national security, falling primarily under consumer safety and regulatory domains.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global consumers
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding generative AI safety and content moderation rather than an active, immediate crisis.
• Autonomy: Human-controlled. The AI system operates as a tool, generating images based on direct user uploads and inputs.
• Novelty: Evolved capability. Demonstrated a significant advancement in the accessibility and scale of synthetic media generation, bypassing basic safety filters.

Impact by dimension

• Physical security: Negligible. No threats to physical infrastructure, kinetic systems, or human physical safety were reported.
• Information security: Negligible. No evidence of state-sponsored information warfare, intelligence compromise, or targeted operations against national security institutions.
• Sovereignty: Negligible. No disruption to state authority, electoral systems, or core government decision-making processes.
• Economic security: Negligible. While the incident raised copyright concerns for digital artists, it did not threaten national economic stability, critical supply chains, or strategic industries.
• Societal stability: Minor. The app exhibited systematic racial and gender biases and generated explicit content, including non-consensual sexualized imagery and child-related content, representing minor societal and human rights concerns manageable under existing legal frameworks.