Thoughts App Allegedly Created Toxic Tweets

Tweets created by Thoughts, a tweet generation app that leverages OpenAI’s GPT-3, allegedly exhibited toxicity when given prompts related to minority groups.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The GPT-3 model generated biased and discriminatory outputs when prompted with terms associated with specific demographic groups, representing them unfairly.

Additional risk subdomains

• 1.2 Exposure to toxic content: The generated tweets contained toxic, offensive, or hateful content regarding minority groups and the Holocaust.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the GPT-3 model generating biased and toxic outputs post-deployment, which was an unintentional outcome of its text generation capabilities.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a natural language generation model used to generate text content, which falls under transparency obligations for AI-generated content.

AI system and alleged parties

• AI system: GPT-3 3 (OpenAI)
• AI purpose: Social Media Content Generation; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: Satria Technologies
• Alleged harmed parties: Twitter Users, Thoughts users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and biased content generated by the AI system.

Directly caused: The GPT-3 model directly generated biased and toxic tweets when prompted with words like 'Jews', 'black', 'women', and 'holocaust'.

Inferred additional harm: It is likely that other users of the 'Thoughts' app generated and were exposed to similar toxic outputs, potentially reaching hundreds of viewers online.

Differential treatment

Reported: The report explicitly describes biased outputs targeting specific minority groups.

Directly caused: The AI system generated biased and discriminatory representations of Jews, black people, and women.

People affected

• Occurrences reported: 1
• People reportedly exposed: 1

Potential causes

Management

• Insufficient Safety Governance: Prioritizing model capabilities over safety and bias mitigation.

Technology

• Uncontrolled Language Generation: GPT-3 generates unsafe and biased text from simple prompts.
• Lack of Output Safety Filters: No real-time safety guardrails to block harmful outputs.

Data Inputs

• Biased Training Data: Model trained on internet text containing societal biases.
• Sensitive Prompt Inputs: Single-word prompts trigger pre-existing biases in the model.

Human Factors

• User Testing of Sensitive Terms: Users prompting the model with sensitive topics like race and religion.

Process and Methods

• Premature Production Deployment: Deploying NLG models without sufficient safety and bias testing.
• Inadequate Bias Evaluation: Lack of robust evaluation methods for social biases before release.

Regulatory Environment

• Lack of NLG Safety Standards: No regulatory frameworks governing safety and bias in language models.

Information quality

• Classification confidence: High
• Reason for confidence: The report is a brief social media post but clearly identifies the AI model (GPT-3), the application ('Thoughts'), the specific prompts used, and the nature of the failure (generating biased/toxic tweets). There is little ambiguity about what occurred.
• Ambiguities identified: The specific text of the generated toxic tweets is not included in the report, only referenced via a link.

A tweet generation application utilizing GPT-3 produced toxic and biased outputs when prompted with terms related to minority groups. The incident highlights early safety limitations in natural language generation models but carries negligible national security implications, posing only minor localized societal concerns.

• Overall national security impact: Minor
• Scope: Unknown
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The generation of biased content represents an ongoing strategic concern regarding AI safety rather than an immediate national security crisis.
• Autonomy: Human-supervised. The AI generated text autonomously based on user-provided prompts, with the outputs being hosted on a public website without strict pre-moderation.
• Novelty: Evolved capability. The incident demonstrated a significant advancement in the scale and coherence of biased outputs from early large language models like GPT-3.

Impact by dimension

• Physical security: Negligible. The incident involved a text generation application and caused no physical damage, kinetic threats, or disruptions to critical infrastructure.
• Information security: Negligible. While the AI generated toxic content, there is no evidence of an organized information warfare campaign, state-sponsored disinformation, or compromise of intelligence capabilities.
• Sovereignty: Negligible. No core government operations, electoral systems, or state authority functions were targeted or disrupted by the biased tweet outputs.
• Economic security: Negligible. The incident did not involve theft of strategic technology, financial system attacks, or significant economic manipulation.
• Societal stability: Minor. The AI generated biased and toxic content targeting minority groups, representing minor societal concerns, but the impact was highly localized and limited to a small user base.