DALL-E 2 Reported for Gender and Racially Biased Outputs

Developers of OpenAI's DALL-E 2 cited risks of the model, varying from misuse as disinformation and explicit content generation, to gender and racial bias.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The DALL-E 2 model generated images that reinforced harmful racial and gender stereotypes, unfairly representing various demographic groups in professional roles.

Additional risk subdomains

• 7.4 Lack of transparency or interpretability: OpenAI's covert modification of user prompts lacked transparency, drawing criticism from researchers regarding the secrecy of their bias mitigation methods.
• 6.3 Economic and cultural devaluation of human effort: The reports highlight concerns that the model's efficiency in generating images could displace jobs for designers, photographers, and artists.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The biased image generation is an unintentional outcome of the AI system's training, occurring post-deployment during its public preview phase.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is an AI-generated content tool (text-to-image generator). Under the EU AI Act, 'AI-generated content such as deepfakes' and systems generating synthetic content fall under Limited Risk, requiring transparency obligations to inform users that the content is AI-generated.

AI system and alleged parties

• AI system: None named
• AI purpose: Image Generation; Visual Art Generation
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI
• Alleged harmed parties: underrepresented groups, Minority Groups

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment in the form of representational bias and disparate performance against marginalized groups.

Directly caused: DALL-E 2 generated images that reinforced gender and racial stereotypes, such as depicting lawyers predominantly as white men and flight attendants as women.

Indirectly caused: Marginalized users faced a performance disparity, bearing the burden of writing highly specific prompts to achieve diverse representation, whereas the default system favored Western, white-passing defaults.

Inferred additional harm: Widespread deployment of such biased models could systematically reinforce societal inequalities and perpetuate stereotypes at scale.

Civil rights

Reported: The report describes potential impacts on human and civil rights through representational harms, stereotyping, and the indignity of erasure.

Indirectly caused: The model's biased outputs and the filtering of prompts related to marginalized groups (e.g., disabilities) can result in the indignity of erasure and exclusion from public discourse.

Inferred additional harm: Continued use of biased generative models could undermine civil rights by normalizing discriminatory representations in media and commercial applications.

Epistemic

Reported: The report discusses the potential for image generators to be used for disinformation and to erode public trust in information systems.

Indirectly caused: The existence of photorealistic synthetic images can reduce the credibility of real images, a phenomenon known as the 'liar's dividend'.

People affected

• Occurrences reported: 1
• People reportedly exposed: 400

Potential causes

Management

• Competitive Pressures: Market competition motivates releasing biased models before resolving safety.
• Sticking Plaster Technical Fix: Management chose a superficial fix instead of systemic cultural change.
• Exclusive Early Access: Releasing to a selected network creates unfair first-mover advantages.

Technology

• Inherited Dataset Bias: Model inherits prejudices from training data scraped from the internet.
• Covert Prompt Modification: System covertly injects words to alter prompts without user knowledge.
• Reference Collisions: Single words with multiple meanings cause unintended explicit generations.

Data Inputs

• Biased Training Data: Training data overrepresents white-passing and male-passing demographics.
• Flawed Training Data Filters: Filtering sexual content inadvertently reduced the generation of women.
• Missing Local Contexts: Training data lacks diverse international and local cultural representations.

Human Factors

• Target Harassment Risks: Users can exploit image modification capabilities to harass individuals.
• Bypassing Safe Filters: Users use visual synonyms and coded language to bypass prompt safety filters.
• Trust in Fake Evidence: Believable synthetic images reduce public trust in overall information systems.

Process and Methods

• Lack of Transparency: Secrecy in model development prevents public assessment of bias mitigations.
• Missing Bias Standards: The AI industry lacks standard metrics or norms to measure and solve bias.
• Imperfect Prompt Classifiers: Prompt filters fail when neutral words are combined in harmful ways.

Regulatory Environment

• Absence of Hard Regulation: No legislative frameworks enforce safety standards in AI deployment.
• Relying on Soft Regulation: Voluntary best practices fail to create repeatable, industry-wide standards.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide detailed, multi-perspective accounts of DALL-E 2's biases, OpenAI's technical mitigations (prompt injection), and the system card details. The facts are well-documented by both independent researchers and OpenAI's own public disclosures.
• Ambiguities identified: The exact algorithmic weights used by OpenAI to modify prompts based on world population are not fully disclosed.
• Alternative interpretations: Some might view the prompt modification as a successful mitigation rather than an incident, but the lack of transparency and the underlying bias make it a safety/fairness incident.

The incident involved OpenAI's DALL-E 2 model exhibiting racial and gender biases during its preview phase, and the subsequent implementation of covert prompt modifications to address the issue. The national security impact is minor, primarily touching on societal stability and human rights through the potential reinforcement of stereotypes, with no direct threats to critical infrastructure, sovereignty, or intelligence.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Multiple nations
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The issue represents an ongoing strategic concern regarding algorithmic bias and transparency rather than an immediate national security crisis.
• Autonomy: Human-supervised. The AI system generates images based on user prompts, but OpenAI implemented automated, system-level prompt modifications without direct user intervention.
• Novelty: Evolved capability. Algorithmic bias is an established issue, but covert prompt manipulation to force diversity represents an evolved mitigation capability.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic safety threats were involved in this text-to-image model bias incident.
• Information security: Negligible. While image generation has potential for disinformation, this specific incident involved representational bias and did not compromise intelligence operations or constitute information warfare.
• Sovereignty: Negligible. There was no impact on state authority, territorial control, electoral systems, or core government decision-making processes.
• Economic security: Negligible. The incident involved a limited-access research preview and did not threaten financial stability, strategic industries, or critical supply chains.
• Societal stability: Minor. The model exhibited representational bias and reinforced gender and racial stereotypes, which could perpetuate societal inequalities if deployed at scale, though impact was limited by restricted access.