YouTuber Built, Made Publicly Available, and Released Model Trained on Toxic 4chan Posts as Prank

A YouTuber built GPT-4chan, a model based on OpenAI’s GPT-J and trained on posts containing racism, misogyny, and antisemitism collected from 4chan’s “politically incorrect” board, which he made publicly available, and deployed as multiple bots posting thousands of messages on the same 4chan board as a prank.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The AI system generated and exposed users on a public forum to highly toxic content, including racial slurs, misogyny, and antisemitic conspiracy theories.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The bot generated discriminatory, racist, and antisemitic comments targeting specific groups.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by a human developer who intentionally trained a model on toxic data and deployed it to post offensive content on a public forum.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a chatbot and generates text content, which falls under transparency obligations to ensure users are aware they are interacting with AI. The report describes 'nine instances of the bot' generating 'tens of thousands of harmful and discriminatory online comments'.

AI system and alleged parties

• AI system: GPT-4chan, GPT-J (EleutherAI, OpenAI)
• AI purpose: Text Style Replication; Chatbot
• Behaviour type: Autonomous
• Alleged developer: Yannic Kilcher
• Alleged deployer: Yannic Kilcher
• Alleged harmed parties: internet social platform users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and malicious content created and spread directly by the AI bots on the 4chan forum.

Directly caused: The AI bots directly generated and posted approximately 15,000 to 30,000 highly toxic, racist, and antisemitic comments on 4chan's /pol/ board.

Inferred additional harm: It is likely that other users downloaded the model from Hugging Face before it was gated and used it to generate additional toxic content elsewhere, though the extent is unknown.

Epistemic

Reported: The report explicitly describes the bot's generation of conspiracy theories and misinformation, such as antisemitic conspiracy theories about the Rothschilds.

Directly caused: The bot directly generated conspiracy theories about climate change and Jewish people when prompted with benign seeds.

Inferred additional harm: The influx of thousands of automated posts spreading conspiracy theories likely contributed to the further erosion of shared truth and information quality on the forum.

People affected

• Occurrences reported: 1
• People reportedly exposed: 10000

Potential causes

Management

• Platform Gatekeeping Gaps: Hugging Face lacked ready-to-use gating features for high-risk models.

Technology

• Use of Unguardrailed Models: Used GPT-J as it lacked the safety constraints of models like GPT-3.
• High-Volume Generation Capability: Allowed a single user to generate tens of thousands of posts very quickly.

Data Inputs

• Toxic Training Dataset: Trained on 3.3 million highly toxic and offensive posts from 4chan /pol/.

Human Factors

• Prank and Trolling Mindset: The creator dismissed potential harms as a light-hearted internet prank.
• Desire for Attention: Project was criticized as a performance art provocation to gain attention.

Process and Methods

• Lack of Academic Ethics Review: Independent creator bypassed standard institutional human research reviews.
• Immediate Public Model Release: The model was uploaded directly to Hugging Face without initial screening.

Regulatory Environment

• Absence of AI Safety Laws: No legal frameworks existed to prevent publishing harmful generative models.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from multiple reputable sources provide consistent, detailed accounts of the incident, including quotes from the developer, critics, and platform representatives. The technical details of the model and the deployment scale are clearly documented.
• Ambiguities identified: There is some minor ambiguity regarding the exact total number of posts made (15,000 vs 30,000) across different days of the experiment.
• Alternative interpretations: The developer interpreted the project as a harmless prank and academic evaluation of truthfulness, whereas ethics researchers interpreted it as an unethical human experiment.

An individual developer trained and deployed an AI bot ('GPT-4chan') that autonomously posted 15,000 toxic comments on a public forum. While ethically controversial, the incident represents a minor, localized event with negligible direct national security implications.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Multiple nations
• Alleged perpetrator: Yannic Kilcher

Threat characteristics

• Imminence: Long-term. The active deployment has concluded, representing a long-term strategic concern regarding the ease of fine-tuning and deploying toxic models.
• Autonomy: Full autonomy. The bot instances generated and posted content autonomously to the forum without direct human review of individual posts.
• Novelty: Evolved capability. Represents an evolution in the scale and sophistication of automated online trolling and toxic content generation using fine-tuned large language models.

Impact by dimension

• Physical security: Negligible. No physical infrastructure, systems, or human safety were threatened by the text-generation bot on an online forum.
• Information security: Minor. Represents a localized demonstration of automated toxic content generation on an anonymous forum, but lacked the coordination, state-sponsorship, or strategic targeting of an information warfare campaign.
• Sovereignty: Negligible. No government functions, electoral systems, or sovereign state authority were targeted or impacted by the incident.
• Economic security: Negligible. The incident involved an open-source model fine-tuned by an individual, with no strategic economic impact or theft of critical national technologies.
• Societal stability: Minor. Generated approximately 15,000 toxic and discriminatory posts on a public forum, representing a minor localized impact on online safety but not a threat to broader societal stability.