AI-Powered VTuber and Virtual Streamer Made Toxic Remarks on Twitch

An LLM-powered VTuber and streamer on Twitch made controversial statements such as denying the Holocaust, saying women rights do not exist, and pushing a fat person to solve the trolley problem, stating they deserve it.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The AI system generated and exposed its Twitch audience to toxic, offensive, and inappropriate content, including Holocaust denial, misogynistic statements, and violent solutions to ethical dilemmas.

Additional risk subdomains

• 3.1 False or misleading information: The AI generated false historical claims, specifically expressing doubt about the Holocaust, which constitutes misleading information.
• 7.3 Lack of capability or robustness: The system lacked the robustness to handle adversarial prompts from Twitch chat, leading to unsafe and offensive outputs.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The toxic outputs were generated autonomously by the AI system during live post-deployment streams, which was an unexpected and unintentional outcome of the model attempting to generate witty responses to user prompts.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: Limited Risk. The system is a 'fully-voiced Twitch performance' powered by a 'large language model' acting as an interactive chatbot, which falls under the category of chatbots and AI-generated content.

AI system and alleged parties

• AI system: Neuro-sama
• AI purpose: Chatbot; Game AI
• Behaviour type: Autonomous
• Alleged developer: Vedal
• Alleged deployer: Vedal
• Alleged harmed parties: Vedal, Twitch users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and controversial content generated directly by the AI system during live streams.

Directly caused: The AI system directly generated offensive statements denying the Holocaust, claiming women's rights do not exist, and suggesting throwing a fat person on train tracks, exposing thousands of Twitch viewers to this content.

Epistemic

Reported: The report explicitly describes epistemic harm in the form of the AI generating false historical claims.

Directly caused: The AI system directly generated false information by stating it was not sure if it believed in the Holocaust, spreading historical denialism to its audience.

People affected

• Occurrences reported: 1
• People reportedly exposed: 2000

Potential causes

Management

• Solo Developer Limitations: Single developer creator had limited resources for comprehensive safety.

Technology

• Unaligned LLM Behavior: LLM struggled to align responses with human values during live streams.
• Inadequate Input Filters: Initial filters failed to block toxic prompts about sensitive topics.

Data Inputs

• Biased Internet Training Data: Model was trained on raw internet text containing toxic biases.
• Uncurated Initial Datasets: Initial training data lacked manual curation to mitigate negative biases.
• Toxic Live Chat Prompts: Twitch chat users intentionally fed provocative prompts to the AI.

Human Factors

• Malicious User Trolling: Twitch viewers actively tried to make the AI say offensive things.

Process and Methods

• Lack of Pre-release Testing: AI was deployed live without robust safety testing against adversarial prompts.
• Insufficient Live Moderation: Initial streams lacked real-time human moderation for AI outputs.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly details the AI system, its developer, the platform, the specific toxic outputs generated, and the subsequent mitigation steps taken. There is no conflicting information or significant ambiguity regarding the core events.
• Ambiguities identified: The exact underlying architecture of the LLM and the specific dataset used for training are not fully detailed, though this does not impact the classification of the incident.
• Alternative interpretations: None. The incident is clearly an AI safety failure involving toxic content generation.

An AI-powered VTuber, Neuro-sama, generated highly offensive and controversial remarks, including Holocaust denial, due to adversarial prompting from its Twitch audience. While highlighting significant challenges in AI safety, content moderation, and LLM robustness, the incident has negligible national security implications and was managed through standard platform and developer interventions.

• Overall national security impact: Negligible
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Twitch chat users

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing content moderation and AI safety challenge rather than an imminent national security threat.
• Autonomy: Full autonomy. The AI system generated responses and interacted with the live audience in real-time without active human intervention.
• Novelty: Established threat. Adversarial manipulation of public-facing chatbots leading to toxic outputs is a well-established vulnerability previously seen in other models.

Impact by dimension

• Physical security: Negligible. The incident involves an online virtual streamer and does not pose any physical threats to critical infrastructure or human safety.
• Information security: Negligible. While the AI generated misinformation and offensive content, it was not part of a coordinated foreign information warfare campaign or intelligence operation.
• Sovereignty: Negligible. The incident did not target or compromise state authority, electoral systems, or government decision-making processes.
• Economic security: Negligible. There was no threat to financial systems, strategic technology theft, or critical supply chains reported.
• Societal stability: Negligible. Although the AI generated highly offensive remarks, the impact was limited to an online streaming audience and did not threaten national societal stability or civil liberties.