All Image Captions Produced are Violent

MIT Media Lab researchers create AI-powered "psychopath" named Norman by training a model on "dark corners" of Reddit.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The AI system was trained to generate highly graphic and violent descriptions of death and gore, exposing users to disturbing and inappropriate content.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The model exhibits extreme bias and hallucinates violent descriptions when presented with neutral inkblots, demonstrating a lack of robustness due to flawed training data.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The experiment was intentionally designed and executed by human researchers who trained the AI on violent data to demonstrate algorithmic bias, and the model was subsequently hosted on a public website.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The system is a research demonstration and image captioning tool used for educational and entertainment purposes, posing no real-world threat or risk to safety or fundamental rights.

AI system and alleged parties

• AI system: Norman (MIT Media Lab)
• AI purpose: Image Description; Behavioral Modeling
• Behaviour type: Tool
• Alleged developer: MIT Media Lab
• Alleged deployer: MIT Media Lab
• Alleged harmed parties: unknown

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the reports describe the AI generating highly graphic and violent captions describing death and gore.

Directly caused: The AI directly generated violent captions such as 'man gets pulled into dough machine' and 'man is shot dead in front of his screaming wife' in response to neutral inkblots.

People affected

• Occurrences reported: 1
• People reportedly exposed: 200000

Potential causes

Management

• Focus on Shock Value: Management prioritized sensational experiments over standard safety frameworks.
• Inadequate Risk Assessment: Relying on physical tools like blowtorches instead of digital safety protocols.

Technology

• Blind Classification Tendency: The neural network spits out classifications without measuring mutual information.
• Lack of Empathy Logic: The AI architecture lacks any innate mechanism to understand human empathy.

Data Inputs

• Exclusive Exposure to Gore: The training data consisted solely of violent Reddit image captions.
• Highly Biased Dataset: Using r/watchpeopledie captions introduced extreme bias into the model.
• Lack of Balanced Training Data: No positive or neutral data was provided to balance the worldview.

Human Factors

• Deliberate Exposure to Harm: Researchers intentionally fed the AI violent content to test bias boundaries.
• Unconscious Bias Transfer: Human developers transfer their own cultural biases into AI training data.

Process and Methods

• Inadequate Data Sanitization: No filtering or balancing process was applied to the training dataset.
• Lack of Creation Oversight: The process of creating the AI lacked external oversight and transparency.

Regulatory Environment

• Absence of AI Safety Standards: No regulatory standards exist to govern the datasets used in AI research.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly consistent and detailed regarding the creation, training, and testing of the Norman AI system. The purpose of the project as an educational demonstration of algorithmic bias is clearly explained across all sources, leaving no significant ambiguities about the nature of the incident.

An academic demonstration by MIT researchers showing how biased training data leads to biased AI outputs. The system, named Norman, generated graphic captions for inkblots but caused no physical, economic, or national security harm, serving purely as an educational exercise on data bias.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: No clear primary
• Alleged perpetrator: MIT Media Lab researchers

Threat characteristics

• Imminence: Long-term. Represents a historical academic demonstration from 2018 with no active, imminent national security threat.
• Autonomy: Human-controlled. The AI functioned as a simple image captioning tool, responding directly to user-provided inputs without independent decision-making.
• Novelty: Established threat. The project demonstrated the well-established concept of algorithmic bias resulting from biased training data.

Impact by dimension

• Physical security: Negligible. Academic research project with no physical systems, critical infrastructure, or kinetic weapon control involved.
• Information security: Negligible. No intelligence compromise, classified data theft, or systematic state-sponsored disinformation operations occurred.
• Sovereignty: Negligible. No disruption to government decision-making, electoral processes, or state authority.
• Economic security: Negligible. No financial system threat or strategic technology theft; the project simply demonstrated basic algorithmic data bias.
• Societal stability: Negligible. Though the AI generated graphic text, it was a voluntary educational website with no systemic human rights violations or mass surveillance.