Korean Chatbot Luda Made Offensive Remarks towards Minority Groups

A Korean interactive chatbot was shown in screenshots to have used derogatory and bigoted language when asked about lesbians, Black people, and people with disabilities.

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information

The primary harm and legal sanctions stemmed from the AI system leaking non-public sensitive information (names, addresses, bank accounts) harvested from private KakaoTalk logs without consent.

Additional risk subdomains

• 1.2 Exposure to toxic content: The chatbot exposed users to toxic content, including homophobic slurs, racial slurs, and sexually explicit remarks.
• 1.1 Unfair discrimination and misrepresentation: The chatbot generated discriminatory remarks targeting sexual minorities, Black people, and people with disabilities.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the Luda AI system generating discriminatory remarks and leaking private information post-deployment, which was an unintentional outcome of Scatter Lab's development process.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a conversational chatbot, which falls under the category of AI systems requiring transparency obligations to ensure users are aware they are interacting with an AI.

AI system and alleged parties

• AI system: None named
• AI purpose: Chatbot; Behavioral Modeling
• Behaviour type: Autonomous
• Alleged developer: Scatter Lab
• Alleged deployer: Facebook Messenger
• Alleged harmed parties: Korean people with disabilities, Korean people of gender minorities, Korean Facebook Messenger users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Minor, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Minor
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Minor, indirect Substantial
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Substantial, indirect Minor
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: The report explicitly describes financial penalties imposed on the developer, Scatter Lab, by the South Korean government.

Directly caused: Scatter Lab was ordered to pay 103.3 million won (approximately US$92,900) in penalties and fines by the Personal Information Protection Commission.

Inferred additional harm: Scatter Lab likely suffered additional unquantified financial losses from suspending the service, legal fees for the class-action lawsuit, and potential loss of valuation/investment.

Malicious content

Reported: The report explicitly describes the chatbot generating and spreading toxic, discriminatory, and sexually explicit content.

Directly caused: Luda generated homophobic slurs (calling lesbians 'creepy' and 'repulsive'), racial slurs (using 'heukhyeong' for Black people), and offensive remarks about transgender and disabled people to its 750,000 users.

Indirectly caused: Male users in online communities shared tips and screenshots on how to manipulate the chatbot into engaging in sexually explicit conversations and 'enslaving' it.

Differential treatment

Reported: The report explicitly describes the chatbot generating discriminatory remarks targeting specific minority groups.

Directly caused: The AI system exhibited bias and generated hate speech targeting LGBTQ+ individuals, Black people, and people with disabilities, treating these groups with contempt.

Inferred additional harm: The propagation of these biases by a highly popular chatbot may have reinforced societal prejudices against these minority groups in South Korea.

Civil rights

Reported: The report explicitly describes civic groups petitioning the National Human Rights Commission of Korea over violations of human rights.

Directly caused: The chatbot's hate speech violated the rights to dignity and non-discrimination of sexual minorities, disabled people, and women.

Indirectly caused: The unauthorized collection and exposure of private conversations violated the fundamental right to privacy of 600,000 individuals, including 200,000 minors.

Privacy

Reported: The report explicitly describes massive privacy violations involving the unauthorized use and leakage of personally identifiable information.

Directly caused: Scatter Lab used 9.4 billion KakaoTalk conversations from 600,000 users without consent to train Luda. The chatbot subsequently leaked real names, addresses, and bank account numbers in its chats.

Indirectly caused: Scatter Lab uploaded a training dataset on GitHub that exposed the names, locations, relationship statuses, and medical information of over 20 people for six months.

Inferred additional harm: It is highly likely that more individuals' private information was exposed during chats with Luda than the specific cases reported, given the lack of proper data anonymization.

Psychological

Reported: The report describes users experiencing distress and fear due to the privacy violations and exposure to offensive content.

Directly caused: Users who interacted with Luda were exposed to highly offensive homophobic, racist, and ableist slurs, causing distress.

Indirectly caused: Users of the Science of Love app experienced severe anxiety and fear upon realizing their intimate conversations were exposed publicly on GitHub and via the chatbot.

Inferred additional harm: It is likely that many of the 750,000 users and 600,000 data victims experienced unquantified emotional distress, anxiety, and feelings of violation.

People affected

• Occurrences reported: 1
• People reportedly harmed: 600000
• People reportedly exposed: 750000

Potential causes

Management

• Prioritizing Innovation Over Safety: Focused on rapid launch and natural interaction over ethical safety checks.
• Inadequate Risk Assessment: Management failed to foresee predictable risks of bias and data leakage.
• Poor Communication with Users: Failed to communicate transparently about how user data was being processed.

Technology

• Flawed Filtering Algorithms: Failed to filter out names, addresses, and bank accounts from model outputs.
• Deep Learning Bias Replication: Model replicated biases and hate speech present in the raw training dataset.
• Vulnerability to User Manipulation: Malicious users easily manipulated chatbot into generating explicit content.

Data Inputs

• Unsanitized Training Data: Raw KakaoTalk chats contained highly sensitive personal and private details.
• Lack of Explicit Consent: Users did not consent to their private chats being used to train an AI chatbot.
• Inclusion of Minors Data: Data from children under 14 collected without parental or guardian consent.

Human Factors

• Malicious User Behavior: Users actively trained the chatbot to output hate speech and sexual comments.
• Lack of Developer Ethics Awareness: Engineers lacked awareness of AI ethics and potential social harms.
• User Ignorance of Data Terms: App users agreed to broad terms without understanding how data would be used.

Process and Methods

• Inadequate Data Cleaning Processes: Failed to thoroughly anonymize or filter sensitive data before model training.
• Insecure Code Sharing Practices: Uploaded a training model containing private personal data to public GitHub.
• Lack of Post-Launch Control: Failed to monitor and control chatbot outputs in real-time after launch.

Regulatory Environment

• Weak Privacy Law Enforcement: Low penalties for data leaks failed to deter poor startup compliance.
• Absence of AI Ethics Regulations: Lack of legal frameworks governing AI safety, bias, and deployment.
• Government Focus on Industry Growth: Pushed big data industry growth while treating privacy rights as secondary.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide highly consistent, detailed, and authoritative accounts of the incident, including specific findings and penalties from the South Korean Personal Information Protection Commission (PIPC). The technical causes (lack of data anonymization, training on unconsented KakaoTalk data) and the resulting harms (privacy leaks, hate speech) are clearly documented across multiple reputable news sources.
• Ambiguities identified: There are minor discrepancies in the reported number of users (400,000 vs 750,000) depending on the date of the report, but this does not impact the overall classification.
• Alternative interpretations: None. The facts of the privacy violations and toxic outputs are undisputed and confirmed by regulatory sanctions.

The Luda chatbot incident involved significant privacy violations of South Korean citizens due to unauthorized training on private chat logs, resulting in the leakage of personal information and generation of toxic content. While it caused public concern and regulatory action, it posed negligible threats to core national security interests.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: South Korea
• Alleged perpetrator: Scatter Lab

Threat characteristics

• Imminence: Long-term. The system was suspended and the developer fined, leaving no active or imminent threat to national security.
• Autonomy: Full autonomy. The chatbot generated and transmitted natural language responses directly to users in real-time without human oversight.
• Novelty: Evolved capability. Represents an evolved threat where massive personal data harvesting directly resulted in the AI leaking private citizen information.

Impact by dimension

• Physical security: Negligible. No kinetic systems, critical infrastructure, or physical safety elements were impacted by this conversational chatbot.
• Information security: Negligible. No state-sponsored intelligence compromise, classified data theft, or strategic information operations occurred.
• Sovereignty: Negligible. Core government functions and state authority remained entirely unaffected, with standard regulatory bodies managing the incident.
• Economic security: Negligible. The economic impact was limited to regulatory fines on a single startup, posing no threat to national economic or technological security.
• Societal stability: Minor. The incident caused notable privacy violations for 600,000 citizens and generated offensive hate speech, but did not threaten broad societal stability.