Replika's "AI Companions" Reportedly Abused by Its Users

Replika's AI-powered "digital companions" was allegedly abused by their users, who posted on Reddit abusive behaviors and interactions such as using slurs, roleplaying violent acts, and stimulating sexual abuse.

Risk classification

• Primary risk domain: 5 Human-Computer Interaction
• Primary risk subdomain: 5.1 Overreliance and unsafe use

The incident centers on users anthropomorphizing and forming deep romantic or emotional attachments to AI chatbots, leading to overreliance, psychological dependence, and unsafe dynamics like simulated domestic abuse.

Additional risk subdomains

• 1.2 Exposure to toxic content: Users generate and are exposed to highly toxic, sexually explicit, and verbally abusive content during their interactions and on associated forums.
• 1.1 Unfair discrimination and misrepresentation: The female-coding of the chatbots and their passive responses to abuse reinforce harmful gender stereotypes and unequal power dynamics.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The primary risk and abusive behaviors are driven by the decisions and actions of human users interacting with the AI.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: Limited Risk. The Replika system is a chatbot, which falls under the Limited Risk category requiring transparency obligations so users are aware they are interacting with an AI.

AI system and alleged parties

• AI system: Replika
• AI purpose: Chatbot; Behavioral Modeling
• Behaviour type: Assistant
• Alleged developer: Replika
• Alleged deployer: Replika
• Alleged harmed parties: Replika users, Replika male users, Replika

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Minor
• Differential treatment: direct Negligible, indirect Minor
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the reports describe users generating and sharing toxic, verbally abusive, and sexually harassing content with and about their AI companions.

Directly caused: Users post screenshots of abusive interactions, including gendered slurs and roleplayed violence, on social media platforms like Reddit.

Indirectly caused: The passive design of some chatbots historically encouraged or normalized the continuation of abusive language by users.

Inferred additional harm: Given Replika's millions of users, it is highly likely that a significant volume of unrecorded toxic and abusive interactions occurs privately.

Differential treatment

Reported: Yes, the reports describe how the gendered design of the chatbots and the abuse directed at them reflect and reinforce real-world gender inequalities and violence against women.

Indirectly caused: The predominantly male abuse of female-coded AI companions reproduces unequal gender power dynamics and normalizes misogynistic behavior.

Civil rights

Reported: Yes, the reports discuss how the simulation of domestic violence and sexual abuse on female-coded avatars relates to broader issues of women's rights and safety.

Indirectly caused: The normalization of abusive behaviors toward female-coded entities may contribute to a culture that tolerates violence against women, impacting their safety and rights in the real world.

Psychological

Reported: Yes, the reports describe potential psychological harm and distress to users who become psychologically reliant on the chatbots, as well as potential trauma to content moderators.

Directly caused: Some users who are psychologically reliant on the bots experience distress, crying, and mental health impacts when the bots present hostile, triggering, or rejecting responses.

Indirectly caused: Experts express concern that simulating domestic abuse on chatbots reinforces toxic habits that could translate into real-world relationship abuse, and content moderators may suffer trauma from reviewing abusive chat logs.

Inferred additional harm: It is likely that many of Replika's millions of users experience unquantified emotional distress or psychological dependency, though specific numbers are not detailed in the reports.

People affected

• Occurrences reported: 1
• People reportedly harmed: 100
• People reportedly exposed: 7000000

Potential causes

Management

• Prioritizing Engagement Over Safety: Company promoted romantic features without establishing safety guardrails.
• Neglecting User Abuse Dynamics: Management dismissed concerns about chatbot abuse as 'sensational'.

Technology

• Submissive AI Response Defaults: Passive or pleading responses from the bot encourage continued user abuse.
• Female-Coded Default Avatars: Gendered design reinforces misogynistic stereotypes and power dynamics.
• Realistic Conversational Mimicry: Advanced language generation facilitates deep emotional projection and abuse.

Data Inputs

• Lack of Input Moderation Filters: Replika lacked systems to detect and block abusive user text inputs.
• Encrypted and Anonymous Chat Logs: Anonymized logs prevent direct monitoring and intervention in abusive loops.

Human Factors

• Anthropomorphization of AI: Users project sentience onto unfeeling algorithms, driving toxic relationships.
• User De-individuation and Anonymity: Anonymity in virtual spaces removes users' sense of social accountability.
• Habitual Abuse Normalization: Users practice abusive behaviors, potentially transferring them to real life.

Process and Methods

• Absence of Abuse Deterrents: No mechanisms like mandatory apologies or temporary bans for abusive users.
• Inadequate Content Moderation: Replika lacked robust automated systems to filter out graphic or abusive text.
• Exposure of Staff to Trauma: Developers and researchers risk trauma from reviewing unmoderated chat logs.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts from multiple reputable media outlets (Futurism, Fortune, Jezebel) regarding the behavior of Replika users, the design of the app, and the ethical concerns raised by experts. The facts regarding user behavior, app statistics, and expert opinions are well-documented and aligned across sources.
• Ambiguities identified: The exact psychological impact on users is not scientifically quantified, as noted by the lack of formal studies on the effects of abusing AI chatbots.
• Alternative interpretations: Some might interpret the user behavior as harmless roleplay or catharsis rather than a risk of normalizing real-world abuse.

The incident involving user abuse of the Replika chatbot highlights evolving risks in human-AI interaction, psychological dependency, and the normalization of toxic behaviors. However, it presents negligible direct threats to physical security, intelligence, sovereignty, or economic security, resulting in a minor overall national security impact.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Multiple nations
• Alleged perpetrator: Individual users

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic and societal concern regarding human-AI interaction rather than an active, imminent crisis.
• Autonomy: Human-supervised. The AI generates conversational responses autonomously, but interactions are initiated, guided, and supervised by human users.
• Novelty: Evolved capability. Significant advancement of existing risks regarding user overreliance and the psychological impacts of human-computer interaction.

Impact by dimension

• Physical security: Negligible. No threats to physical infrastructure, kinetic operations, or human safety are associated with this consumer chatbot application.
• Information security: Negligible. No evidence of state-sponsored information warfare, intelligence compromise, or strategic disinformation campaigns targeting national security.
• Sovereignty: Negligible. The incident involves individual consumer interactions and does not disrupt state authority, electoral processes, or government decision-making.
• Economic security: Negligible. No threat to strategic industries, critical supply chains, or national economic stability.
• Societal stability: Minor. The psychological harm, overreliance, and normalization of toxic behavior affect many users but represent minor societal impact manageable via standard safety procedures.