AI Training Dataset for Detecting Nudity Allegedly Found to Contain CSAM Images of Identified Victims

An image dataset, NudeNet, used to train systems for detecting nudity was reportedly found to contain CSAM images, including material involving identified or known victims. According to the Canadian Centre for Child Protection, the dataset had been widely downloaded and cited in academic research prior to discovery. The images were allegedly included without vetting, exposing researchers to legal risk and perpetuating harm to victims. The dataset was subsequently removed following notification.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The NudeNet dataset exposed researchers and academic systems to child sexual abuse material (CSAM), which is a severe form of toxic and illegal content.

Additional risk subdomains

• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: The dataset distributed highly sensitive, non-consensual private images of children, violating their privacy.

Causal factors

• Entity: Human
• Intent: Unintentional
• Timing: Pre-deployment

The inclusion of CSAM in the dataset was caused by human decisions during the data collection phase, which occurred pre-deployment and was entirely unintentional.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The NudeNet system is an image classifier designed for nudity detection, which is a content moderation application analogous to spam filters and poses low risk to society when functioning as intended.

AI system and alleged parties

• AI system: NudeNet classifier
• AI purpose: NSFW Content Detection; Image Classification
• Behaviour type: Tool
• Alleged developer: NudeNet model developers, NudeNet dataset maintainers
• Alleged deployer: Research institutions, Independent researchers, Dataset users, AI researchers, AI developers, Academic researchers
• Alleged harmed parties: minors, Individuals subjected to sexual exploitation imagery, Identified CSAM victims, Academic researchers

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Substantial, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Substantial, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Substantial
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Substantial, indirect Substantial

Malicious content

Reported: The report explicitly describes the presence of over 120 images of child sexual abuse material (CSAM) in the dataset.

Directly caused: The NudeNet dataset directly contained and distributed over 120 images of CSAM.

Inferred additional harm: Other unvetted datasets scraped in similar ways likely contain additional undetected CSAM.

Civil rights

Reported: The report explicitly describes the exploitation of children and distribution of CSAM, which violates fundamental human rights.

Directly caused: The inclusion of CSAM in the dataset directly violates the fundamental rights of the children depicted.

Inferred additional harm: The lack of due diligence in AI training data curation poses a systemic threat to the rights of children globally.

Privacy

Reported: The report explicitly notes that victims did not consent to have their images distributed and used in training.

Directly caused: The dataset distributed highly sensitive, non-consensual private images of children, violating their privacy.

Inferred additional harm: Thousands of individuals' private images may be swept up in unvetted web-scraping datasets.

Psychological

Reported: The report explicitly notes the ethical issue of distributing images of victims who did not consent, which constitutes further exploitation.

Indirectly caused: The distribution of CSAM of known victims causes severe psychological harm and re-traumatization to the victims depicted.

Inferred additional harm: Widespread psychological distress to the identified victims and their families due to the viral distribution of these images in academic datasets.

Child sexual exploitation and abuse

Reported: The report explicitly describes a CSEA incident involving the NudeNet dataset containing CSAM.

Directly caused: The dataset contained over 120 images of CSAM, directly facilitating the distribution of child sexual exploitation material.

Indirectly caused: Academic projects and models trained on this dataset indirectly propagated the exploitation by hosting and utilizing the material.

Inferred additional harm: Additional children may have been affected by undetected CSAM in this or similar scraped datasets.

People affected

• Occurrences reported: 1
• People reportedly harmed: 120
• People reportedly exposed: 250

Potential causes

Management

• Prioritizing Speed Over Safety: Focus on rapid AI development led to skipping safety and ethical reviews.
• Lack of Liability Awareness: Creators ignored legal liabilities of distributing illegal materials.

Technology

• Lack of Automated Filtering: No automated tools like PhotoDNA were run on the dataset before release.
• Indiscriminate Web Scraping: Scraping scripts collected images from the internet without safety filters.

Data Inputs

• Unvetted Training Images: Dataset contained over 120 unvetted, illegal CSAM images.
• Lack of Source Verification: No verification of image provenance or consent for scraped internet data.

Human Factors

• User Overreliance on Datasets: Researchers assumed academic datasets were clean without verifying contents.
• Ethical Blind Spots: In the rush to innovate, researchers ignored potential ethical harms.

Process and Methods

• Inadequate Dataset Vetting: No formal process to review and clean dataset before public hosting.
• Lack of Due Diligence: Creators failed to perform due diligence on scraped image contents.

Regulatory Environment

• Permissive Research Hosting: Platforms like Academic Torrents host datasets without mandatory pre-screening.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, factual details from a reputable organization (C3P) regarding the dataset contents, academic usage, and subsequent removal. There are no conflicting accounts or major ambiguities regarding the core incident.

The discovery of child sexual abuse material within the NudeNet dataset exposed global academic researchers to legal and compliance risks. While representing severe ethical and human rights violations, the direct threat to national security remains minor and is managed through standard law enforcement and child protection channels.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global academic and research institutions
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The dataset has been removed from major platforms, representing an ongoing strategic concern for data curation rather than an active crisis.
• Autonomy: Human-controlled. The incident involves static dataset curation and training data ingestion, which are entirely human-controlled processes.
• Novelty: Established threat. The inclusion of toxic, illegal, or non-consensual data in unvetted web-scraped datasets is a well-documented and established risk in AI development.

Impact by dimension

• Physical security: Negligible. No threats to physical systems, critical infrastructure, or kinetic targeting are associated with this image dataset.
• Information security: Negligible. No active information warfare, state-sponsored disinformation campaigns, or compromise of classified intelligence occurred.
• Sovereignty: Negligible. The incident does not impact state authority, electoral processes, or core government decision-making operations.
• Economic security: Minor. Academic institutions and researchers faced minor legal and criminal liabilities due to hosting illicit material, impacting research continuity.
• Societal stability: Minor. While constituting severe individual human rights violations and privacy breaches of victims, the national security threat to societal stability remains minor and manageable under standard law enforcement procedures.