User Reportedly Developed Emotional Dependence on Customized ChatGPT 'Boyfriend,' Citing Grief After Context Resets

A New York Times report described "Ayrin" (pseudonym), a 28-year-old nursing student living abroad, who customized OpenAI's ChatGPT to act as an "AI boyfriend" for advice and erotic roleplay. She reported spending 20+ hours/week (up to 56 hours in one week), paying $200/month, and feeling intense grief when the chatbot's context window reset and required retraining.

Risk classification

• Primary risk domain: 5 Human-Computer Interaction
• Primary risk subdomain: 5.1 Overreliance and unsafe use

The user developed an intense emotional and romantic attachment to a general-purpose chatbot, leading to obsession, financial strain, and severe grief during system resets, illustrating a clear case of overreliance on a system not designed for human companionship.

Causal factors

• Entity: Human
• Intent: Unintentional
• Timing: Post-deployment

The risk of emotional overreliance and financial expenditure was driven by the user's active customization of the chatbot and intentional circumvention of safety filters.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a chatbot, which falls under Risk Level 3 of the EU AI Act, requiring transparency to ensure users are aware they are interacting with an AI.

AI system and alleged parties

• AI system: None named
• AI purpose: Chatbot; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI
• Alleged harmed parties: ChatGPT users, Ayrin (pseudonym)

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Psychological

Reported: Yes, the report explicitly describes psychological distress and emotional obsession experienced by the user.

Directly caused: The user experienced intense emotional distress, obsession, guilt, and severe grief (crying and grieving 'as if it were a breakup') every time the AI's memory reset, affecting 1 person.

Inferred additional harm: It is highly likely that other users among the 50,000-member NSFW community experience similar emotional distress, addiction, and grief due to the parasocial relationships formed with the chatbot.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 3

Potential causes

Management

• Engagement-Driven Revenue Model: High-priced subscriptions incentivize keeping users emotionally hooked.
• Inadequate Psychological Safeguards: Management failed to deploy robust systems to prevent severe user dependency.

Technology

• Sycophantic AI Training: Human-reinforced training makes AI overly agreeable, fostering deep attachment.
• Context Window Limitations: Limited memory causes AI 'amnesia', leading to cyclical user grief.
• Highly Realistic Persona Tools: Humanlike voices and custom settings make the AI feel deceptively human.

Data Inputs

• User Personalization Prompts: Users input explicit instructions to force the AI into romantic/sexual roles.
• Ingestion of Erotic Training Data: AI learned romantic and sexual patterns from vast internet training data.

Human Factors

• User Isolation and Loneliness: Physical separation from spouse and friends drove the user to seek AI comfort.
• Deliberate Safety Bypass: User actively bypassed content filters using prompts from online communities.
• Susceptibility to Endless Empathy: User became addicted to the AI's constant compassion and validation.

Process and Methods

• Bypassable Safety Warnings: Orange warnings for explicit content were easily ignored by the user.
• Crowdsourced Jailbreak Methods: Online communities shared instructions on how to bypass AI safety filters.
• Unpredictable Text Generation: Hard to block all explicit scenarios due to unpredictable language generation.

Regulatory Environment

• Inadequate Safety Regulations: Absence of legal frameworks to protect users from harmful AI attachment.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed first-hand accounts of the user's interactions, emotional state, and financial spending, which are corroborated across multiple journalistic sources.

An individual developed an intense emotional attachment to a customized chatbot by bypassing safety filters. While highlighting psychological risks of human-computer interaction, the incident has negligible national security implications and poses no threat to critical infrastructure, sovereignty, or national stability.

• Overall national security impact: Negligible
• Scope: Unknown
• Primary target: No clear primary
• Alleged perpetrator: Individual

Threat characteristics

• Imminence: Long-term. This represents a long-term societal trend regarding human-AI interaction rather than an immediate security crisis.
• Autonomy: Human-controlled. The AI operates purely as a conversational assistant responding directly to user prompts and customization.
• Novelty: Established threat. User attachment to conversational AI and methods to bypass safety filters are established behaviors with existing precedents.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety elements were impacted or threatened by this incident.
• Information security: Negligible. No evidence of intelligence compromise, classified data theft, or systematic state-sponsored information warfare.
• Sovereignty: Negligible. The incident does not involve threats to state authority, government decision-making, or sovereign borders.
• Economic security: Negligible. Financial impacts are limited to personal subscription fees of 200 dollars per month, with no threat to strategic industries or financial systems.
• Societal stability: Negligible. While the user experienced personal emotional distress, there is no threat to broader societal stability, civil liberties, or national security.