.avif){kind=small}
In August 2024, South Korea experienced a massive surge in non-consensual sexually explicit deepfake imagery targeting women and minors. Perpetrators, often teenagers, used AI tools to manipulate social media photos of classmates, teachers, and neighbors into pornographic content, distributing them through Telegram chat groups. The incident has caused widespread panic, leading many women to delete their social media presence, and has prompted government investigations and calls for stricter regulation of both AI tools and messaging platforms.
At the end of August 2024, South Korean authorities began investigating a significant surge in the creation and dissemination, often via Telegram, of explicit deepfake pornography created without consent from the stolen social media content of female classmates, teachers, and neighbors.
The incident involves the targeted creation of humiliating and sexually explicit synthetic imagery of specific victims (classmates, teachers, and acquaintances) for harassment and exploitation.
The incident was caused by human perpetrators who intentionally used deployed AI deepfake tools to generate non-consensual sexually explicit images of women and minors.
Limited Risk: The report describes AI-generated content such as deepfakes, which under the EU AI Act are classified as limited risk, requiring specific transparency obligations to ensure users are aware the content is AI-generated.
Severity ratings are based on this scale.
Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.
Reported: The report explicitly describes the generation and distribution of highly toxic and sexually explicit deepfake images and videos.
Directly caused: AI tools were used to generate thousands of non-consensual pornographic deepfake images and videos of women and minors.
Indirectly caused: These explicit images were shared and discussed in Telegram 'humiliation rooms' to facilitate verbal abuse and harassment.
Inferred additional harm: A massive volume of synthetic pornographic content continues to circulate in private, encrypted digital spaces beyond the reach of law enforcement.
Reported: The report explicitly describes systematic differential treatment and targeting of women and girls.
Directly caused: AI deepfake tools were used to systematically target, sexualize, and humiliate female students, teachers, and military personnel.
Indirectly caused: The crisis forced women and girls to self-censor, delete their social media photos, and restrict their online behavior, a burden not shared by their male peers.
Inferred additional harm: The incident reinforces systemic gender-based discrimination and unequal safety expectations for women in digital environments.
Reported: The report explicitly describes violations of victims' rights to dignity, privacy, and freedom from sexual exploitation.
Directly caused: The non-consensual sexualization of women and minors violated their fundamental rights to personal dignity and bodily autonomy.
Indirectly caused: The pervasive threat of deepfakes restricted women's freedom of expression and digital participation on social media platforms.
Inferred additional harm: Widespread, ongoing violations of civil rights and digital safety for female citizens due to inadequate legal protections.
Reported: The report explicitly describes privacy violations through the unauthorized use of personal photos and information.
Directly caused: Personal photos (selfies, school photos) and personal details (names, ages, school locations) were harvested from social media without consent to generate deepfakes.
Indirectly caused: Victims' personal contact details and locations were leaked alongside the explicit images in Telegram chatrooms.
Inferred additional harm: Systematic harvesting of public and semi-private personal data of thousands of women and minors for malicious exploitation.
Reported: The report explicitly describes severe psychological distress, anxiety, panic, and trauma among the victims.
Directly caused: Victims reported feeling 'petrified', 'so alone', and experiencing severe anxiety and trauma, with one victim describing racing heartbeats upon receiving phone notifications.
Indirectly caused: Widespread panic and anxiety spread among young women and schoolgirls across South Korea, leading to hyper-vigilance and fear of male acquaintances.
Inferred additional harm: Thousands of undetected victims likely suffer from unaddressed psychological trauma, anxiety, and depression due to the non-consensual sexualization of their likenesses.
Reported: The report explicitly describes epistemic harm caused by the highly realistic nature of the deepfakes.
Directly caused: AI tools generated highly realistic synthetic images that combined real faces with fake bodies, fabricating false visual evidence of sexual acts.
Indirectly caused: The realism of the deepfakes caused confusion, making it difficult for victims to prove the images were fabricated, with some police initially dismissing cases because 'the photos were fake'.
Inferred additional harm: A broader erosion of trust in digital media and personal photography, as any image can be convincingly manipulated to create false narratives.
Reported: The report explicitly describes the generation and distribution of sexually explicit deepfake images targeting underage schoolgirls.
Directly caused: AI tools were used to generate sexually explicit deepfake images of minors, some of whom were under the age of 16.
Indirectly caused: These explicit images of minors were shared and distributed in Telegram groups, some of which had thousands of members.
Inferred additional harm: Widespread, undetected creation and distribution of synthetic child sexual abuse material (CSAM) targeting school-aged girls.
A massive wave of AI-generated non-consensual sexually explicit deepfakes in South Korea targeted women and minors, distributed via Telegram. The incident caused widespread societal panic, prompt government and police intervention, and highlighted significant human rights, privacy, and digital safety challenges, representing a substantial impact on societal stability.