iGPT, SimCLR Learned Biased Associations from Internet Training Data

Unsupervised image generation models trained using Internet images such as iGPT and SimCLR were shown to have embedded racial, gender, and intersectional biases, resulting in stereotypical depictions.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The unsupervised models mathematically encode and reproduce human stereotypes and gender/racial biases, representing women and minorities in an unfair and stereotypical manner.

Additional risk subdomains

• 1.2 Exposure to toxic content: The models capture internet data containing toxic content and can be used to generate sexualized or pornographic images targeting women.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Pre-deployment

The risk of algorithmic bias is caused by the unsupervised learning process of the AI models, which unintentionally capture societal stereotypes from internet data prior to downstream deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report discusses the potential use of these models in 'policing or hiring, where models are already analyzing candidate video recordings to decide if they are a good fit for the job.' Under the EU AI Act, AI systems used in recruitment (employment) and law enforcement (policing) are classified as High Risk.

AI system and alleged parties

• AI system: SimCLR, iGPT (Google, OpenAI)
• AI purpose: Data Grouping; Image Generation
• Behaviour type: Tool
• Alleged developer: OpenAI, Google
• Alleged deployer: OpenAI, Google
• Alleged harmed parties: underrepresented groups in training data, racial minority groups, gender minority groups

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Minor
• Differential treatment: direct Negligible, indirect Minor
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the report describes how the models encode toxic patterns and can be used to generate sexualized photos of women.

Directly caused: The models directly encode toxic patterns and stereotypical associations from internet data, such as clustering pixel embeddings of women away from professional contexts.

Indirectly caused: The report notes that similar image-generation algorithms have led to an explosion of deepfake pornography targeting women, which these models could facilitate.

Differential treatment

Reported: Yes, the report describes how the models encode gender and racial biases that could lead to differential treatment if deployed.

Directly caused: The models mathematically represent men and women unequally, placing photos of men close to professional items like suits while placing women farther away.

Indirectly caused: The researchers warn that deploying these biased models in hiring or policing could lead to discriminatory decisions against women and minorities.

People affected

• Occurrences reported: 1

Potential causes

Management

• Lack of Corporate Transparency: Companies restrict model access, preventing independent safety investigations.
• Insufficient Dataset Accountability: Organizations fail to take responsibility for auditing training data contents.

Technology

• Unsupervised Pixel Embeddings: Models cluster pixels by co-occurrence, capturing underlying biased patterns.
• Generative Model Exploitation: Algorithms like iGPT can be misused to generate deepfake pornography of women.

Data Inputs

• Biased Internet Data Composition: Web-scraped datasets overrepresent stereotypes and sexualized images.
• Disturbing Historic Labels: Foundational datasets like ImageNet contained slurs and offensive human tags.

Human Factors

• Societal Biases in Internet Content: Human biases are naturally encoded in the online media used for training.

Process and Methods

• Inadequate Model Testing: Lack of pre-deployment testing for biases in downstream computer vision tasks.
• Poor Dataset Curation Standards: Absence of responsible methods to document and curate training datasets.

Regulatory Environment

• Lack of Deployment Standards: No regulatory oversight for using biased models in hiring or policing.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly outlines the academic study, the specific models evaluated (iGPT and SimCLR), and the nature of the biases discovered. Since this is a research paper identifying pre-deployment risks rather than an active real-world failure incident, the lack of direct real-world harm is expected and clearly documented.
• Alternative interpretations: The incident could be interpreted purely as an academic finding rather than an active safety incident, but it represents a critical safety risk evaluation of foundational models.

An academic study revealed that unsupervised computer vision models, specifically iGPT and SimCLR, mathematically encode societal racial and gender biases from their training datasets. As a pre-deployment research finding with no active exploitation or real-world deployment reported, the immediate national security impact is negligible, though it highlights ongoing strategic challenges in AI safety.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The findings represent an ongoing strategic and technical concern regarding algorithmic bias rather than an active, imminent threat.
• Autonomy: Human-controlled. The evaluated AI models function as mathematical tools for image embedding and generation, operating under direct human control without autonomous agency.
• Novelty: Established threat. Algorithmic bias and dataset representation issues are well-documented and established challenges in AI development.

Impact by dimension

• Physical security: Negligible. The incident is an academic study identifying algorithmic bias in computer vision models, presenting no threats to physical systems or critical infrastructure.
• Information security: Negligible. No active information warfare or intelligence compromise occurred, though researchers noted potential downstream risks of deepfake generation.
• Sovereignty: Negligible. There is no indication of threats to state authority, electoral systems, or core government operations in this academic finding.
• Economic security: Negligible. While identifying vulnerabilities in models from US companies, the incident represents a pre-deployment research finding with negligible direct economic or strategic impact.
• Societal stability: Negligible. The study highlights potential civil rights risks if biased models are deployed in policing or hiring, but no actual real-world deployment or societal harm occurred.