Harmful Stereotyping of Non-Cisgendered People via Text-to-Image Systems

Text-to-image systems such as DALL-E are allegedly generating biased and often insulting representations of non-cisgender identities. The systems tend to generate stereotypical and sexualized images when prompted with gender identity terms like "trans," "nonbinary," or "queer," highlighting systemic issues of bias.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The incident involves text-to-image AI systems generating stereotypical, sexualized, and dehumanizing representations of non-cisgender and indigenous identities, resulting in unfair representation of these groups.

Additional risk subdomains

• 1.3 Unequal performance across groups: The AI system's performance degrades when prompted with non-cisgender terms, producing less human-looking and lower-quality images compared to standard prompts.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The biased and stereotypical image generation is an action of the deployed AI system (DALL-E) that occurs post-deployment and is an unintentional outcome of the model's training on societal data.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes text-to-image generators, which fall under AI-generated content. According to the definitions, 'AI-generated content such as deepfakes' and systems requiring transparency obligations represent Limited Risk (Level 3).

AI system and alleged parties

• AI system: DALL-E (OpenAI)
• AI purpose: Image Generation; Visual Art Generation
• Behaviour type: Tool
• Alleged developer: OpenAI
• Alleged deployer: DALL-E
• Alleged harmed parties: Non-cisgender individuals, LGBTQ+ community

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and biased content created directly by the AI system.

Directly caused: The AI system directly generated stereotypical, sexualized, and dehumanizing images of non-cisgender and indigenous people.

Inferred additional harm: Widespread generation and potential online dissemination of sexualized and dehumanized depictions of queer and indigenous identities, potentially reaching thousands of users.

Differential treatment

Reported: The report explicitly describes differential treatment of prompts containing non-cisgender terms compared to standard prompts.

Directly caused: The AI system treats prompts containing non-cisgender terms differently by generating lower-quality, highly stereotyped, and sexualized images.

Indirectly caused: Proposed mitigation strategies (like ignoring terms or adding warning labels) would further marginalize and treat queer identities as taboo.

Inferred additional harm: Systematic bias in text-to-image models leads to unequal representation and treatment of marginalized groups at scale across all users of these systems.

Civil rights

Reported: The report explicitly describes concerns regarding data ownership and the right to fair representation without being exoticized or dehumanized.

Directly caused: Dehumanizing and sexualizing representations of non-cisgender and indigenous (two-spirit) people violate their right to dignity and fair representation.

Inferred additional harm: Broad-scale deployment of biased models undermines the civil rights of marginalized groups by reinforcing harmful societal prejudices and stereotypes.

Psychological

Reported: The report explicitly describes psychological distress and negative feelings among non-cisgender individuals regarding their misrepresentation and proposed mitigation strategies.

Directly caused: The survey of 35 non-cisgender people showed they felt strongly, negatively, and impassioned about being misrepresented or having their identities made invisible/taboo by proposed mitigations.

Indirectly caused: Widespread feelings of alienation, being stereotyped, and dehumanized among the broader queer community exposed to these AI-generated images.

Inferred additional harm: It is likely that many more non-cisgender individuals experience distress, anxiety, and a sense of marginalization when encountering these stereotypical and sexualized AI-generated representations.

People affected

• Occurrences reported: 1
• People reportedly harmed: 35
• People reportedly exposed: 35

Potential causes

Technology

• Statistical pattern matching: Models detect and reproduce societal biases rather than understanding concepts.
• Flawed prompt response: Adding identity terms leads to less human and more sexualized images.

Data Inputs

• Biased training datasets: Training data contains societal transphobia and stereotypes of queer people.
• Exotified representation of minorities: Indigenous data depicts two-spirit people only in religious dress.
• Data ownership concerns: Unresolved ownership issues with images of indigenous and marginalized groups.

Human Factors

• Societal transphobia: Prejudices embedded in society are captured in text and images online.

Process and Methods

• Exclusion of queer community: Solutions are designed without discussion with the affected community.
• Heuristic mitigation strategies: Warning labels and omitting terms make identities feel taboo or invisible.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, academic-backed details about the specific biases in text-to-image models, quoting researchers and survey results. The AI system (DALL-E) is explicitly identified, and the nature of the harm (stereotyping, sexualization, community alienation) is well-documented.
• Ambiguities identified: The exact version of DALL-E or other text-to-image models tested is not specified, and the exact prompts used are represented by general terms rather than full strings.
• Alternative interpretations: The incident could be viewed purely as an academic study of bias rather than an active 'incident' of harm, but the real-world impact on the queer community's perception and representation justifies treating it as an active safety risk.

A research study highlights systemic biases in text-to-image AI models like DALL-E, which generate stereotypical, dehumanizing, and sexualized images of non-cisgender and indigenous individuals. While representing a significant ethical and societal concern regarding discrimination, the direct national security implications remain minor.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic and technical challenge regarding algorithmic bias rather than an active, time-sensitive crisis.
• Autonomy: Human-controlled. The AI system operates as a tool generating images based on direct user text prompts.
• Novelty: Established threat. Algorithmic bias and representation issues in machine learning models are well-documented, established challenges in AI development.

Impact by dimension

• Physical security: Negligible. No physical security, critical infrastructure, or kinetic threats are associated with this text-to-image bias incident.
• Information security: Negligible. The incident involves algorithmic bias in image generation rather than active information warfare, intelligence compromise, or state-sponsored disinformation operations.
• Sovereignty: Negligible. There is no indication of threats to state authority, electoral processes, or core government decision-making systems.
• Economic security: Negligible. The incident highlights algorithmic limitations and bias in commercially available models, but does not involve strategic technology theft or threats to critical economic infrastructure.
• Societal stability: Minor. The models produce highly biased, stereotypical, and sexualized depictions of marginalized communities, representing systemic discrimination but without direct threats to national societal stability.