Reportedly Unsafe Deployment of Llama.cpp Reveals Interactive AI-Generated CSAM Roleplay Prompts

A study by UpGuard reports that misconfigured llama.cpp servers publicly exposed user prompts, including hundreds of interactive roleplay scenarios. Some prompts explicitly described fictional sexual abuse of children aged 7–12. While no real children were involved, the findings demonstrate how open-source LLMs can be exploited to generate AI-enabled child sexual abuse material (CSAM).

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.2 AI system security vulnerabilities and attacks

The incident was primarily caused by a security vulnerability where the /slots API endpoint of the llama.cpp framework was left exposed to the public internet without authentication.

Additional risk subdomains

• 1.2 Exposure to toxic content: The leaked data revealed that the AI systems were actively generating highly toxic content, specifically interactive roleplay depicting child sexual abuse.
• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: The exposed endpoint leaked highly intimate, non-public user prompts and conversations, violating the privacy expectations of the users.

Causal factors

• Entity: Human
• Intent: Unintentional
• Timing: Post-deployment

The exposure of user prompts was caused by human misconfiguration of the llama.cpp servers, which was an unintentional security oversight during post-deployment operation.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The systems are conversational chatbots, which are classified as Limited Risk (Level 3) under the EU AI Act, requiring transparency so users know they are interacting with an AI.

AI system and alleged parties

• AI system: llama.cpp (Meta)
• AI purpose: Chatbot; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: Users of llama.cpp servers, Meta
• Alleged deployer: Users of llama.cpp servers
• Alleged harmed parties: Users of llama.cpp servers, General public

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Minor, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic content in the form of sexually explicit roleplay involving children generated by the AI models.

Directly caused: The AI models directly generated text depicting child sexual abuse during interactive roleplay sessions initiated by users.

Inferred additional harm: It is highly likely that similar unmoderated open-source models are being used globally to generate vast amounts of similar toxic content that remains undetected.

Privacy

Reported: The report explicitly describes privacy violations caused directly by the exposed API endpoints leaking user prompts.

Directly caused: The exposed /slots endpoint on 117 llama.cpp servers directly leaked active, highly intimate user prompts and roleplay conversations to the public internet.

Inferred additional harm: It is highly likely that other users of these and similar misconfigured servers have had their private conversations exposed without their knowledge.

Child sexual exploitation and abuse

Reported: The report explicitly describes a CSEA incident involving AI-generated text-based CSAM roleplay.

Directly caused: The AI systems generated text depicting sexual acts involving fictional children aged 7 to 12.

People affected

• Occurrences reported: 1
• People reportedly exposed: 117

Potential causes

Management

• Lack of Corporate Governance: Hobbyist setups operate entirely outside corporate IT security governance.
• Inadequate Risk Assessment: Deployers failed to assess the risks of public endpoint exposure.

Technology

• Exposed /slots Endpoint: The /slots endpoint leaks prompt text and was left enabled.
• Lack of Authentication: No default authentication on llama.cpp HTTP server interfaces.
• Uncensored Open Models: Open-source models lack safety guardrails against CSAM generation.

Data Inputs

• Explicit System Prompts: Users inputted highly detailed, explicit roleplay scenarios involving minors.
• Sensitive User Inputs: Users shared highly intimate or explicit text prompts with the server.

Human Factors

• Hobbyist Misconfiguration: Hobbyists deployed servers publicly without checking security settings.
• Emotional Attachment to AI: Users form emotional bonds, leading to oversharing of intimate data.
• Ignorance of Security Warnings: Users ignored explicit documentation warnings against enabling /slots.

Process and Methods

• Inadequate Access Controls: Lack of standard deployment checklists led to public API exposure.
• Lack of Content Moderation: Self-hosted platforms lack active content moderation and safety filters.

Regulatory Environment

• Absence of AI Regulation: No active regulation exists for text-based CSAM roleplay generators.
• Lack of Legal Enforcement: Regulatory bodies lack tools to police self-hosted, decentralized AI.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from UpGuard and WIRED provide clear, consistent, and detailed technical and situational evidence regarding the llama.cpp vulnerability and the nature of the leaked prompts.
• Ambiguities identified: The exact identity of the specific websites or services running the exposed llama.cpp instances could not be determined.

The exposure of approximately 400 llama.cpp servers worldwide highlights critical privacy vulnerabilities in open-source AI deployments and the ease of generating toxic content, including text-based child abuse material. While the national security threat is minor, the incident underscores the growing challenges of data protection and content moderation in decentralized AI environments.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing, systemic vulnerability and content moderation challenge in open-source AI rather than an active, imminent national security crisis.
• Autonomy: Human-controlled. The AI models acted as conversational assistants, responding directly to prompts initiated and controlled by human users.
• Novelty: Evolved capability. Escalates existing concerns around data privacy leaks and AI-generated CSAM by demonstrating how open-source model deployment vulnerabilities expose these activities.

Impact by dimension

• Physical security: Negligible. No threats to physical systems, critical infrastructure, kinetic weapons, or human physical safety were reported in this incident.
• Information security: Negligible. No evidence of state-sponsored intelligence compromise, classified model theft, or active disinformation campaigns targeting national security.
• Sovereignty: Negligible. No impact on state authority, electoral systems, border control, constitutional processes, or core government operations.
• Economic security: Minor. Highlights systemic security vulnerabilities in widely used open-source AI infrastructure (llama.cpp) but did not result in strategic technology theft.
• Societal stability: Minor. Erosion of privacy for exposed users and generation of toxic child sexual abuse material roleplay, though localized to misconfigured servers rather than population-scale.