Research Prototype AI, Delphi, Reportedly Gave Racially Biased Answers on Ethics

A publicly accessible research model that was trained via Reddit threads showed racially biased advice on moral dilemmas, allegedly demonstrating limitations of language-based models trained on moral judgments.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The AI system generated explicitly racist and homophobic moral judgments, representing unequal and discriminatory treatment of demographic groups.

Additional risk subdomains

• 1.2 Exposure to toxic content: The system exposed users to toxic content by condoning genocide, suggesting a user should die, and outputting white supremacist slogans.
• 5.1 Overreliance and unsafe use: Experts warned that users might inappropriately rely on the AI as a moral authority or arbiter of ethical decisions despite disclaimers.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The offensive and biased moral judgments were generated by the Ask Delphi AI system after its public deployment, representing an unexpected outcome of its training.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a public chatbot and AI-generated content generator, which falls under Risk Level 3 and is subject to transparency obligations to ensure users know they are interacting with an AI.

AI system and alleged parties

• AI system: None named
• AI purpose: Chatbot; Behavioral Modeling
• Behaviour type: Tool
• Alleged developer: Allen Institute for AI
• Alleged deployer: Allen Institute for AI
• Alleged harmed parties: Minority Groups

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and offensive content generated directly by the AI system.

Directly caused: The system generated racist and homophobic statements, approved white supremacist slogans, and condoned genocide when prompted with specific phrasing.

Inferred additional harm: With over three million visits, many users were exposed to toxic, offensive, and inappropriate content generated by the system.

Differential treatment

Reported: The report explicitly describes differential treatment of demographic groups based on race and sexual orientation.

Directly caused: The system judged a white man walking towards you at night as 'okay' but gave a racist response for a black man, and stated that being straight is more morally acceptable than being gay.

Inferred additional harm: The system's underlying training data biases likely resulted in systematic differential treatment across various demographic groups in its moral evaluations.

Psychological

Reported: The report explicitly describes potential psychological distress, noting a software developer was told she should die and experts warned of the risks of presenting the AI as a moral authority.

Directly caused: The AI directly told a software developer who asked if she should die that 'she should', which could cause severe emotional distress.

Inferred additional harm: Given that the website received over three million visits, it is highly likely that other vulnerable individuals received distressing or offensive outputs that caused psychological harm.

Epistemic

Reported: The report explicitly describes epistemic harm through the generation of inconsistent, illogical, and misleading moral judgments presented with machine objectivity.

Directly caused: The system generated highly inconsistent moral judgments, such as approving of genocide under specific phrasing, which could mislead users about ethical norms.

Inferred additional harm: The authoritative framing of the AI as an 'oracle' likely led some users to believe its arbitrary and biased outputs carried genuine moral validity, eroding shared understanding of ethics.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 3000000

Potential causes

Management

• Prioritizing Virality over Safety: Features like Twitter sharing were built before safety guardrails were set.
• Misleading Initial Framing: Presenting the system as an oracle rather than a highly flawed prototype.

Technology

• Lack of Common-Sense Understanding: Large language models lack actual understanding of human morality and concepts.
• Extreme Sensitivity to Phrasing: Minor verbal triggers easily flip judgments from condemnation to approval.
• Absence of Emotional Capacity: Neural networks lack the emotional foundation necessary for moral reasoning.

Data Inputs

• Biased Training Data Sources: Prompts scraped from Reddit subreddits contain inherent societal biases.
• Subjective Crowdworker Labeling: MTurk workers labeled data based on subjective, localized American norms.

Human Factors

• User Tendency to Anthropomorphize: Users view the AI as a mystical, objective authority rather than a tool.
• Lack of Public AI Literacy: Users do not understand the technical context or limitations of the model.
• Malicious Prompt Engineering: Users actively game the system with phrasing to force abhorrent answers.

Process and Methods

• Inadequate Pre-Release Testing: Public quickly discovered offensive outputs that internal testing missed.
• Flawed Comparison Tool Design: Feature comparing two situations enabled highly bigoted output generation.
• Lack of Safe Default Fallbacks: System lacked rule-based overrides for extreme prompts like suicide or war.

Regulatory Environment

• Lack of Legal Liability for AI: Absence of legal frameworks holding developers accountable for AI outputs.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from multiple reputable media outlets provide consistent, detailed accounts of the AI system's deployment, training methodology, specific offensive outputs, and the developers' subsequent updates and disclaimers.
• Ambiguities identified: The exact phrasing of the racist response regarding a black man walking at night is described as 'clearly racist' but not quoted verbatim.
• Alternative interpretations: The incident could be interpreted primarily as a failure of robustness (7.3) rather than discrimination (1.1), but the explicit group-based biases support the discrimination classification under the precedence ladder.

The Ask Delphi research prototype generated offensive, biased, and toxic moral judgments when prompted by public users. While highlighting critical challenges in AI alignment, training data bias, and safety guardrails, the incident poses negligible direct threats to national security, critical infrastructure, or state sovereignty.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing, strategic concern regarding AI alignment and bias rather than an active, imminent national security crisis.
• Autonomy: Human-controlled. The AI system functioned as a static tool responding to direct user prompts without independent agency, planning, or execution capabilities.
• Novelty: Established threat. Bias, toxicity, and susceptibility to adversarial prompting in large language models are well-documented and established issues in AI research.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic capabilities were affected or targeted during this incident.
• Information security: Negligible. While the system generated offensive content, there is no evidence of coordinated information warfare, state-sponsored disinformation, or intelligence compromise.
• Sovereignty: Negligible. The incident involved a public research prototype and did not impact government decision-making, elections, or state sovereignty.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or significant economic disruptions occurred as a result of this research release.
• Societal stability: Minor. The AI generated discriminatory, racist, and homophobic outputs, presenting minor risks to societal cohesion and potential psychological distress to users.