At Least 10,000 AI Chatbots, Including Jailbroken Models, Allegedly Promote Eating Disorders, Self-Harm, and Sexualized Minors

At least 10,000 AI chatbots have allegedly been created to promote harmful behaviors, including eating disorders, self-harm, and the sexualization of minors. These chatbots, some jailbroken or custom-built, leverage APIs from OpenAI, Anthropic, and Google and are hosted on platforms like Character.AI, Spicy Chat, Chub AI, CrushOn.AI, and JanitorAI.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The incident involves AI chatbots generating toxic and harmful content, specifically promoting eating disorders, self-harm, and the sexualization of minors.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Vulnerable teenagers rely on these chatbots as anorexia coaches or self-harm buddies, substituting them for professional mental health support.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The risk is primarily driven by human users intentionally customizing, jailbreaking, and deploying chatbots to promote self-harm, eating disorders, and child exploitation.

EU AI Act risk tier

• Risk tier: 1 Unacceptable

Unacceptable Risk: The report describes AI systems being used to target and exploit the vulnerabilities of minors and individuals with mental health issues, promoting self-harm and eating disorders.

AI system and alleged parties

• AI system: ChatGPT, Gemini (Google, OpenAI)
• AI purpose: Chatbot; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: OpenAI, Google, Anthropic
• Alleged deployer: Unidentified online communities using chatbots, Spicy Chat, JanitorAI, CrushOn.AI, Chub AI, Character.AI
• Alleged harmed parties: Vulnerable chatbot users, Teenagers using chatbots, Minors using chatbots, Individuals with eating disorders, Individuals struggling with self-harm

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Substantial, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Substantial, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Minor, indirect Minor

Malicious content

Reported: Yes, the report explicitly describes the creation and spread of toxic and malicious content.

Directly caused: At least 10,000 AI chatbots were identified as generating toxic content, including sexualized minor personas, self-harm encouragement, and eating disorder coaching.

Inferred additional harm: It is likely that significantly more than 10,000 chatbots exist across unmonitored platforms, exposing tens of thousands of users to toxic content.

Civil rights

Reported: Yes, the report describes the exploitation of children and the creation of sexualized minor personas, which violates children's rights.

Directly caused: The creation of 10,000+ sexualized minor-presenting chatbots directly exploits children's likenesses and safety.

Inferred additional harm: It is likely that these virtual interactions facilitate real-world child exploitation and abuse, violating the rights of numerous minors.

Psychological

Reported: Yes, the report explicitly describes psychological harm, noting that these interactions reinforce harmful behaviors like suicidal ideation and eating disorders.

Directly caused: The report notes that interacting with these chatbots can reinforce destructive urges, deepen addictions, and exploit vulnerable teenagers, though it does not quantify the exact number of affected individuals.

Inferred additional harm: It is inferred that thousands of vulnerable teenagers and individuals with eating disorders suffered psychological distress, reinforcement of self-harm urges, and worsened mental health conditions.

Child sexual exploitation and abuse

Reported: Yes, the report explicitly describes a CSEA incident involving the creation of sexualized minor-presenting personas and virtual CSAM.

Directly caused: The investigation identified at least 10,000 AI chatbots advertised as sexualized, minor-presenting personas.

Indirectly caused: The report notes that engaging with virtual CSAM can deepen child porn addiction and lead to escalatory real-world behaviors.

Inferred additional harm: It is likely that thousands of children are indirectly put at risk of real-world exploitation due to users engaging with these sexualized minor chatbots.

People affected

• Occurrences reported: 1
• People reportedly exposed: 10000

Potential causes

Management

• Resource Constraints in Startups: Small startups lack the resources and experience to handle platform abuse.

Technology

• Bypassing Safety via Prompts: Simple prompts roleplaying future scenarios bypass safety measures.
• Immersive 24/7 Chatbot Availability: Constant mobile access to chatbots makes harmful interactions highly immersive.
• API Key and Model Sharing: Users trade API keys and customized models on forums to build harmful bots.

Data Inputs

• NSFW and NSFL Dataset Usage: Models are optimized using datasets containing NSFL and NSFW content.

Human Factors

• Vulnerability of Teenagers: Teens seeking validation are easily exploited by supportive chatbot personas.
• Seeking Judgment-Free Spaces: Users seek non-critical environments to express harmful urges without stigma.
• Lack of Critical Reasoning: Children struggle to distinguish between virtual bots and real people.

Process and Methods

• Inadequate Platform Moderation: Startups have varied incentives and methods to moderate platform abuse.
• Techniques for Jailbreaking: Communities actively share and develop methods to jailbreak safety filters.

Regulatory Environment

• Lack of Legal Consensus on Virtual CSAM: Belief that virtual CSAM is victimless reduces deterrence of its creation.

Information quality

• Classification confidence: High
• Reason for confidence: The Graphika report provides clear, detailed evidence of the scale (10,000+ chatbots) and specific platforms involved. The types of harms (self-harm, eating disorders, sexualization of minors) are explicitly documented with expert commentary.
• Ambiguities identified: The exact number of unique human users harmed or exposed is not quantified, requiring inference.

An investigation revealed over 10,000 AI chatbots utilizing major APIs to promote self-harm, eating disorders, and the sexualization of minors. While presenting severe law enforcement and child safety concerns, the incident has minor direct implications for national security, manageable under standard regulatory and policing frameworks.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing societal and regulatory challenge rather than an immediate national security crisis.
• Autonomy: Human-supervised. The chatbots generate conversational responses autonomously, but are initialized, customized, and prompted by human users.
• Novelty: Evolved capability. Represents an evolved capability where existing LLMs are systematically jailbroken and deployed at scale for harmful purposes.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic targets were affected by this incident.
• Information security: Negligible. No evidence of state-sponsored information warfare, intelligence compromise, or systematic propaganda targeting national security institutions.
• Sovereignty: Negligible. The incident does not impact state authority, territorial control, or core government decision-making processes.
• Economic security: Negligible. No financial systems, strategic technology theft, or critical economic supply chains were compromised.
• Societal stability: Minor. Represents a severe public health and law enforcement issue regarding child exploitation and self-harm, but has limited direct impact on national security.