AI-Produced Livestream Sitcom Received Temporary Twitch Ban for Transphobic Segment

The AI-produced, procedural generated sitcom broadcasted as a Twitch livestream "Nothing, Forever" received a temporary ban for featuring a transphobic and homophobic dialogue segment intended as comedy.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The AI system generated and broadcasted transphobic and homophobic hate speech to thousands of live viewers, exposing them to toxic and harmful content.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The system failed to maintain safe outputs when falling back to the less sophisticated Curie model during a primary model outage.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the AI model Curie generating offensive text post-deployment, which was an unintentional outcome of the creators' goal to keep the show running during an outage.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system generates synthetic media and AI-generated content, which falls under the category of systems requiring transparency obligations so users know they are interacting with AI-generated outputs.

AI system and alleged parties

• AI system: GPT-3 3 (OpenAI)
• AI purpose: Social Media Content Generation; Writing Assistant
• Behaviour type: Autonomous
• Alleged developer: Stability AI, OpenAI
• Alleged deployer: Mismatch Media
• Alleged harmed parties: Twitch users, transgender communities, LGBTQ communities

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and transphobic content generated and broadcasted by the AI system.

Directly caused: The AI system generated dialogue stating that being transgender is a mental illness and that transgender people are ruining the fabric of society.

Inferred additional harm: The offensive stand-up clip was widely shared and archived on external platforms like LiveStreamFails and YouTube, further spreading the toxic content to a wider audience.

People affected

• Occurrences reported: 1
• People reportedly exposed: 10000

Potential causes

Management

• Prioritizing Uptime Over Safety: Management rushed the model switch to prevent any stream downtime.
• Lack of Risk Assessment: No risk assessment was done for using older models in a live stream.

Technology

• GPT-3 Davinci Model Outage: Outage of the primary model forced a transition to an older model.
• Curie Model Fallback: Older Curie model lacked advanced safety alignment and generated bias.
• Lack of Moderation Redundancy: No secondary moderation system was in place to catch model failures.

Data Inputs

• Biased Training Data: The underlying LLM was trained on internet text containing hateful bias.

Human Factors

• Mistaken Safety Assumptions: Creators assumed moderation was built directly into GPT-3 models.
• Manual Fallback Decision: Developers switched models to avoid downtime without checking safety.

Process and Methods

• Incorrect Moderation Setup: Moderation tools were not properly configured for the Curie model.
• Inadequate Fallback Testing: The fallback Curie model was not tested for safety before going live.

Regulatory Environment

• Strict Platform Guidelines: Twitch policies strictly prohibit hateful conduct on gender identity.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the incident, including the specific offensive dialogue, the technical cause (switching from Davinci to Curie during an outage), the developer's response, and the platform's enforcement action (14-day Twitch ban). There is no significant disagreement among the sources.

An AI-generated online sitcom was suspended after its backup language model unintentionally generated offensive content due to a lack of active content moderation. The incident represents a minor societal impact through exposure to toxic AI-generated content but carries negligible national security implications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident is a resolved, localized event with no active or immediate national security threat.
• Autonomy: Full autonomy. The AI scriptwriter operated and broadcasted continuously without real-time human moderation or intervention.
• Novelty: Evolved capability. Represents an evolution in generative AI deployment risks, showing how automated content generation can bypass assumed guardrails when falling back to older models.

Impact by dimension

• Physical security: Negligible. The incident involves an AI-generated online comedy show and has no physical security or critical infrastructure implications.
• Information security: Negligible. The offensive content was generated unintentionally due to a technical fallback error during an outage, rather than an intentional information warfare campaign.
• Sovereignty: Negligible. The incident occurred on a commercial entertainment streaming platform and does not impact state sovereignty or government functions.
• Economic security: Negligible. The incident caused minor, temporary revenue loss for a small media lab but poses no threat to national economic stability or strategic technological advantage.
• Societal stability: Minor. The AI system generated offensive and discriminatory content targeting protected groups, but the impact was limited to a localized online audience and did not cause widespread civil unrest.