South Korean man used AI to create sexual images of children

A South Korean man used AI technology to generate 360 images of a sexual nature depicting children in April of 2023. The police confiscated the images and the courts sentenced the man to two and a half years in prison. The case marked the first of its nature in the South Korean court system.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The incident involved the generation of child sexual abuse material (CSAM), which is explicitly classified under exposure to toxic content.

Additional risk subdomains

• 4.3 Fraud, scams, and targeted manipulation: The report also highlights cases in Spain and on Twitch where AI was used to create non-consensual sexual imagery to target, extort, or humiliate specific victims.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by a human user who intentionally used a deployed AI system to generate sexually exploitative images of children.

EU AI Act risk tier

• Risk tier: 1 Unacceptable

Risk Level 1: Unacceptable Risk. The system was used to generate child sexual abuse material, which represents an exploitative system targeting vulnerabilities and violating fundamental rights, posing significant dangers to society.

AI system and alleged parties

• AI system: AI technology
• AI purpose: Deepfake Image Generation; Image Generation
• Behaviour type: Tool
• Alleged developer: unknown
• Alleged deployer: Unnamed South Korean man
• Alleged harmed parties: General public

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Minor, indirect Negligible

Malicious content

Reported: The report explicitly describes the creation of toxic and malicious content in the form of AI-generated child sexual abuse material.

Directly caused: The perpetrator directly generated approximately 360 sexually exploitative images of children using AI.

Inferred additional harm: It is highly likely that similar AI tools are being used globally to generate vast amounts of undetected toxic and sexually explicit content depicting minors.

Civil rights

Reported: The report mentions that AI can be used to violate people's bodily autonomy and safety, especially for women and minors.

Indirectly caused: The generation of child sexual abuse material represents a fundamental violation of children's rights and societal safety norms.

Child sexual exploitation and abuse

Reported: The report explicitly describes a CSEA incident involving the generation of child sexual abuse material.

Directly caused: A South Korean man used AI to generate 360 sexually exploitative images of children.

Inferred additional harm: It is likely that other individuals are using similar generative AI technologies to create and potentially distribute CSAM, posing a severe ongoing risk to child safety.

People affected

• Occurrences reported: 1

Potential causes

Management

• Prioritizing Growth Over Safety: AI developers release powerful tools without adequate safety guardrails.

Technology

• High-Level Realistic AI Generation: AI technology creates highly realistic images of non-existent minors.
• Deepfake Generation Tools: Accessible tools allow easy manipulation of images to create explicit content.
• Lack of Algorithmic Safeguards: Generative models lacked built-in blocks to prevent generating explicit minors.

Data Inputs

• Target Images Used Without Consent: Real photos of victims inputted into AI tools to generate explicit images.
• Unrestricted Training Data: AI models trained on datasets that enable highly realistic child synthesis.

Human Factors

• Malicious Intent of Users: Users deliberately leverage AI to create abusive and exploitative content.
• Lack of Ethical Constraints: Perpetrators disregard consent and legal boundaries of victims.

Process and Methods

• Inadequate Platform Moderation: Streaming and sharing platforms fail to proactively block deepfakes.
• Delayed Policy Enforcement: Platforms update policies reactively only after major public controversies.

Regulatory Environment

• Lagging AI Regulations: Governments struggle to keep pace with rapid generative AI advancements.
• Ambiguous Legal Definitions: Unclear laws regarding whether virtual child imagery constitutes abuse.
• Lack of Global Standards: Inconsistent enforcement and rules across different jurisdictions.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly details the legal case, the number of images generated (360), the sentence (2.5 years), and the fact that they were not distributed. While the specific AI model is not named, the nature of the incident as AI-generated CSAM is unambiguous.
• Ambiguities identified: The specific AI model or platform used to generate the images is not identified in the report.

A South Korean court sentenced an individual to prison for generating AI-based child sexual abuse material, establishing a novel legal precedent that realistic virtual depictions of minors constitute CSAM. While representing a significant legal milestone for AI regulation and human rights protection, the incident was successfully managed by local law enforcement and poses negligible threats to broader national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: South Korea
• Alleged perpetrator: Individual

Threat characteristics

• Imminence: Long-term. The immediate criminal threat has been resolved through prosecution, though the strategic challenge of regulating AI-generated CSAM remains a long-term concern.
• Autonomy: Human-controlled. The AI system acted purely as a tool, executing image generation commands directly inputted by the human user.
• Novelty: First-of-its-kind. Marks the first prosecution and legal precedent in South Korea defining AI-generated virtual depictions of minors as child sexual abuse material.

Impact by dimension

• Physical security: Negligible. No threats to physical infrastructure, kinetic systems, or physical human safety were reported in this incident.
• Information security: Negligible. No information warfare, state-sponsored disinformation, or intelligence compromise occurred; the generated images were confiscated prior to distribution.
• Sovereignty: Negligible. This was a domestic criminal case handled within the standard judicial system, posing no threat to state sovereignty or core government functions.
• Economic security: Negligible. The incident involved an individual user and did not impact strategic technologies, financial systems, or national economic security.
• Societal stability: Minor. The case establishes a legal precedent regarding virtual child exploitation, representing a minor societal impact managed within standard judicial and law enforcement procedures.