Bad AI-Written Christmas Carols

Janelle Shane, an AI research scientist, used 240 popular Christmas carols to train a neural network to write its own carols. This incident has been downgraded to an issue as it does not meet current ingestion criteria.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The primary theme across the downgraded incidents is the identification of AI system limitations, safety vulnerabilities, and performance failures under experimental or academic testing conditions.

Additional risk subdomains

• 2.2 AI system security vulnerabilities and attacks: Incident 159 specifically details security research exposing adversarial vulnerabilities in Tesla's Autopilot lane recognition system.
• 1.2 Exposure to toxic content: Incident 287 describes a GPT-3 medical chatbot experiment where the system generated highly toxic advice by telling a mock patient to kill themselves.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Pre-deployment

The risks and failures described in the reports occurred during testing, academic research, or experimental phases prior to any actual real-world deployment or harm.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes AI systems used in critical safety-relevant applications such as Tesla's Autopilot system and healthcare applications like Nabla's GPT-3 medical chatbot experiment, which fall under high-risk categories due to safety implications.

AI system and alleged parties

• AI system: GPT-3
• AI purpose: Chatbot; Autonomous Driving
• Behaviour type: Assistant
• Alleged developer: Janelle Shane
• Alleged deployer: Janelle Shane
• Alleged harmed parties: Carollers

Harm severity

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 6
• People reportedly exposed: 10

Potential causes

Management

• Premature Tech Evaluation: Attempting to use raw GPT-3 for medical tasks without proper fine-tuning.

Technology

• Inherent LLM Inconsistency: GPT-3 lacks deterministic logic, leading to unsafe medical advice.
• Lack of Medical Reasoning: The model cannot verify scientific accuracy or medical safety rules.

Data Inputs

• Unfiltered Training Data: Training on web text includes toxic and unsafe self-harm references.
• Lack of Specialized Medical Data: General-purpose training lacks clinical safety alignment.

Human Factors

• User Prompting of Sensitive Topics: Mock patients testing safety limits triggered extreme model responses.

Process and Methods

• Insufficient Safety Guardrails: No real-time output filtering to block self-harm suggestions.
• Inadequate Clinical Validation: Testing was done in an experimental setup without clinical safety gates.

Regulatory Environment

• Lack of Healthcare AI Standards: No formal regulations governing LLM deployment in clinical triage.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly outlines the reasons why each incident was downgraded, explicitly stating that no real-world harm occurred and classifying them as academic, experimental, or projected vulnerabilities. This makes the assessment of zero real-world harm highly confident.
• Ambiguities identified: The report is a meta-compilation of multiple distinct incidents, which makes classifying them under a single primary domain challenging, but the common thread is the lack of real-world harm and presence of capability/robustness limitations.
• Alternative interpretations: One could analyze each of the six incidents individually, which would yield different domain classifications (e.g., security for Tesla, toxic content for GPT-3).

This meta-compilation details six historical AI cases reclassified as 'issues' due to a complete lack of real-world harm. The cases span academic research, simulated experiments, and early-stage system limitations, presenting negligible overall impact on national security.

• Overall national security impact: Negligible
• Response level: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. These historical and academic cases represent theoretical vulnerabilities and experimental failures rather than imminent security threats.
• Autonomy: Human-supervised. The systems analyzed, including autonomous driving and language models, operate under human supervision or within controlled experimental frameworks.
• Novelty: Established threat. The issues highlight well-documented, early-stage limitations in AI capabilities, robustness, and adversarial vulnerability.

Impact by dimension

• Physical security: Negligible. The Tesla Autopilot vulnerability was identified during controlled security research by Tencent Keen Security Lab, resulting in no real-world kinetic or critical infrastructure impact.
• Information security: Negligible. The GPT-3 Guardian op-ed was an edited experimental piece rather than an active information warfare or state-sponsored disinformation campaign.
• Sovereignty: Negligible. None of the reported incidents involved threats to state authority, electoral systems, or core government decision-making processes.
• Economic security: Negligible. The incidents represent academic evaluations and early-stage experimental failures, presenting no threat to strategic industries or national economic security.
• Societal stability: Negligible. While Nabla's medical chatbot experiment and Harvard's FaceTag app raised ethical and privacy questions, they occurred in simulated environments without causing real-world harm.