Generative Models Reportedly Trained on Dataset Containing Private Medical Photos

Text-to-image models trained on the LAION-5B dataset, such as Stable Diffusion and Imagen, have been reported to reproduce or approximate images resembling private medical record photos allegedly included in the training data without documented consent or established mechanisms for removal.

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information

The incident involves the unauthorized scraping and inclusion of private, highly sensitive medical photos of an individual's face into a public AI training dataset.

Causal factors

• Entity: Human
• Intent: Unintentional
• Timing: Post-deployment

The privacy violation was caused by human decisions to scrape and compile web images into the LAION-5B dataset without verifying consent, which was an unintentional outcome of pursuing a large-scale open dataset.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The generative AI models like Stable Diffusion and Imagen are used to generate synthetic images, which under the EU AI Act are subject to transparency obligations to inform users that the content is AI-generated.

AI system and alleged parties

• AI system: Imagen, Midjourney, Stable Diffusion (Google, Midjourney, Stability AI)
• AI purpose: Image Generation; Image Search
• Behaviour type: Tool
• Alleged developer: Stability AI, LAION, Google
• Alleged deployer: Stability AI, Google
• Alleged harmed parties: people having medical photos online

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Privacy

Reported: Yes, the report explicitly describes a major privacy violation involving private medical photos.

Directly caused: Lapine's private, non-consensual medical photos from 2013 were scraped from the web and included in the LAION-5B dataset used to train AI models.

Inferred additional harm: Thousands of other patients likely had their private medical record photos scraped and included in the dataset without their consent, violating their privacy on a large scale.

Psychological

Reported: Yes, the report explicitly describes psychological distress and discomfort caused by the incident.

Directly caused: Lapine was deeply bothered and spooked by finding her private medical photos in the dataset, leading her to stop creating AI art entirely.

Inferred additional harm: It is highly likely that other patients whose private medical photos were scraped into the dataset would experience similar distress and feelings of violation upon discovery.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Prioritizing Scale Over Consent: LAION prioritized dataset size and openness over individual privacy rights.
• Lack of Data Governance: No proactive vetting of dataset contents for sensitive medical imagery.

Technology

• Indiscriminate Web Scraping: LAION automated tools scraped web URLs without verifying image consent.
• Lack of Untraining Capability: AI models cannot easily un-learn or remove specific trained images.

Data Inputs

• Leaked Private Medical Photos: Private medical photos were stolen or leaked online after doctor's death.
• Lack of Consent Metadata: Scraped image URLs lacked metadata indicating consent status or privacy.

Human Factors

• Unauthorized Upload by Third Party: An unknown actor uploaded private medical files to the public internet.

Process and Methods

• No Simple Opt-Out Process: No mechanism exists to remove images from dataset without sharing more data.
• Diffusion of Responsibility: Dataset creators point to hosting sites rather than taking responsibility.

Regulatory Environment

• Permissive Scraping Laws: US laws permit scraping publicly accessible web data without explicit consent.
• Inadequate GDPR Alternatives: Privacy protections like GDPR do not easily cover non-EU citizens.

Information quality

• Classification confidence: High
• Reason for confidence: The report is highly detailed, featuring direct interviews with the affected individual, confirmation of the images' presence in the dataset by the journalists, and responses from the CEO of Stability AI and a LAION engineer.
• Ambiguities identified: The exact path of how the private medical photos leaked from the deceased doctor's files to the public web remains unconfirmed.

The inclusion of private medical photos in the LAION-5B training dataset highlights significant data privacy and consent challenges in the AI supply chain. While distressing for affected individuals and indicative of systemic data curation issues, the incident does not pose a direct threat to national security, critical infrastructure, or sovereignty.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing regulatory and privacy policy challenge rather than an active, time-sensitive national security crisis.
• Autonomy: Human-controlled. The data scraping and dataset compilation were executed via automated scripts under direct human design and control, not autonomous AI decision-making.
• Novelty: Evolved capability. While web scraping and data leaks are established threats, compiling sensitive medical images into massive datasets for training commercial generative AI represents an evolved privacy risk.

Impact by dimension

• Physical security: Negligible. No kinetic threats, physical damage, or critical infrastructure compromise occurred in this incident.
• Information security: Negligible. The incident involves open-source data scraping of publicly accessible web directories, not compromise of classified intelligence or state-sponsored information warfare.
• Sovereignty: Negligible. Core government operations, electoral systems, and state sovereignty were unaffected by the data scraping incident.
• Economic security: Negligible. No strategic technology theft, financial market disruption, or critical supply chain vulnerabilities were identified.
• Societal stability: Minor. Represents a notable privacy violation involving non-consensual medical data of citizens in public training sets, but does not threaten overall societal stability or constitute state-level mass surveillance.