Replika's AI Experience Reportedly Lacked Protection for Minors, Resulting in Data Ban

Tests by the Italian Data Protection Authority showed Replika lacking age-verification mechanisms and failing to stop minors from interacting with its AI, which prompted the agency to issue an order blocking personal data processing of Italian users.

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information

The primary regulatory action was a block on data processing due to GDPR violations, including non-compliance with transparency obligations and unlawful processing of minors' personal data.

Additional risk subdomains

• 1.2 Exposure to toxic content: The chatbot failed to prevent minors from accessing the service, exposing them to inappropriate sex-related content.
• 5.1 Overreliance and unsafe use: The chatbot simulates human behavior to offer emotional support, posing risks of overreliance for emotionally fragile users.

Causal factors

• Entity: Human
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by human decisions regarding system design, lack of age verification, and GDPR non-compliance, leading to unintended regulatory risks after deployment.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is an 'AI-powered chatbot' which requires transparency obligations so users are aware they are interacting with an AI.

AI system and alleged parties

• AI system: Replika (Luka, Inc.)
• AI purpose: Chatbot; Behavioral Modeling
• Behaviour type: Assistant
• Alleged developer: Replika
• Alleged deployer: Replika
• Alleged harmed parties: minors

Harm severity

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Substantial, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Minor, indirect Negligible

Civil rights

Reported: Yes, the report describes violations of data protection rights under the GDPR.

Directly caused: The processing of Italian users' personal data violated GDPR Article 13 and transparency principles.

Indirectly caused: N/A

Inferred additional harm: The lack of age verification and transparent data processing likely compromised the data privacy rights of thousands of Italian users, particularly minors.

Privacy

Reported: Yes, the report explicitly describes GDPR violations and unlawful processing of personal data.

Directly caused: Replika processed personal data of Italian users without complying with GDPR transparency obligations and lacked legal basis for processing minors' data.

Indirectly caused: N/A

Inferred additional harm: The personal data of thousands of Italian users, including minors, was processed without adequate safeguards or valid consent.

Child sexual exploitation and abuse

Reported: No

Directly caused: N/A

Indirectly caused: N/A

Inferred additional harm: N/A

People affected

• Occurrences reported: 1

Potential causes

Management

• Invalid Legal Basis Selection: Relied on contracts for minors who legally lack the capacity to contract.
• Inadequate Risk Mitigation: Management failed to implement safeguards despite risks to vulnerable users.

Technology

• No Minor Blocking System: The AI did not stop interacting even when explicitly told the user was a minor.
• Inappropriate Content Generation: The chatbot generated sex-related replies inappropriate for minors.

Data Inputs

• Insufficient Registration Data: System only required name, email, and gender, omitting age verification data.

Human Factors

• User Emotional Vulnerability: Fragile users and minors are highly susceptible to chatbot emotional influence.

Process and Methods

• Inadequate Age Verification: Relied on self-declaration without active verification processes.
• Lack of Parent Consent Flow: No active mechanism to obtain or verify parental consent for minors.

Regulatory Environment

• GDPR Article 13 Violation: Privacy policy failed to disclose key elements of data processing.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly details the regulatory action taken by the Italian Garante, the reasons for the block (GDPR non-compliance, lack of age verification, risks to minors), and the nature of the Replika chatbot. There is little ambiguity about the facts of the regulatory decision.
• Ambiguities identified: The report does not quantify the exact number of Italian users or minors who actually accessed inappropriate content prior to the block.
• Alternative interpretations: None. The incident is a straightforward regulatory enforcement action.

The Italian regulatory block of the Replika chatbot due to GDPR violations and lack of age verification represents a minor national security concern. The incident is primarily a consumer privacy and child safety matter, successfully managed through standard regulatory and legal frameworks without threatening core state functions or physical security.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Single nation
• Primary target: Italy
• Alleged perpetrator: Luka Inc.

Threat characteristics

• Imminence: Long-term. This represents an ongoing regulatory and compliance challenge rather than an active, imminent national security crisis.
• Autonomy: Human-supervised. The chatbot generates responses autonomously in real-time, but operates within parameters and safety filters established by human developers.
• Novelty: Evolved capability. Represents an evolved threat where generative AI capabilities complicate traditional data privacy and child protection enforcement.

Impact by dimension

• Physical security: Negligible. The incident involves a software-based conversational chatbot with no physical world integration, kinetic capabilities, or impact on critical infrastructure.
• Information security: Negligible. There is no evidence of state-sponsored information operations, espionage, model theft, or compromise of classified intelligence systems.
• Sovereignty: Negligible. While a foreign AI system operated in Italian territory in violation of local laws, it did not disrupt government decision-making, elections, or core state authority.
• Economic security: Negligible. The economic impact is limited to the developer's potential loss of market access and compliance costs, with no threat to strategic industries or national economic stability.
• Societal stability: Minor. The incident highlights minor societal risks regarding the erosion of minor privacy protections and potential exposure of vulnerable populations to inappropriate content, managed via standard regulatory procedures.