Apparent Failure to Accurately Label Primates in Image Recognition Software Due to Alleged Fear of Racial Bias

Eight years after Google Photos mislabeled images of Black individuals as "gorillas," image recognition software by Google, Apple, Amazon, and Microsoft still shows signs of either avoiding or inaccurately categorizing primates. Tests reveal that Google and Apple Photos refrain from labeling primates altogether, possibly to avoid the risk of perpetuating racial stereotypes. Microsoft OneDrive fails to identify any animals, while Amazon Photos overgeneralizes in its labeling.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The AI system performed with lower accuracy for Black individuals due to biased training data, leading to unequal outcomes and offensive misclassifications.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The system's output mislabeled Black individuals as primates, perpetuating harmful and offensive racial stereotypes.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The racist mislabeling was an unexpected and unintentional outcome of the AI system's image classification process after it was deployed to the public.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The consumer photo-tagging and search applications described in the report pose minimal risk to safety or fundamental rights under the EU AI Act, falling under Risk Level 4.

AI system and alleged parties

• AI system: Amazon Photos, Apple Photos, Google Photos, Microsoft OneDrive (Amazon, Apple, Google, Microsoft)
• AI purpose: Image Classification; Image Search
• Behaviour type: Tool
• Alleged developer: Microsoft, Google, Apple, Amazon
• Alleged deployer: Microsoft, Google, Apple, Amazon
• Alleged harmed parties: members of racial and ethnic minorities who risk being stereotyped or misrepresented, Consumers relying on accurate image categorization

Harm severity

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Minor
• Civil rights: direct Substantial, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: Yes, the report describes how the system performed poorly specifically for individuals with darker skin tones.

Directly caused: The image recognition software failed to accurately identify Black individuals, misclassifying them as primates while performing accurately for lighter-skinned individuals.

Indirectly caused: Tech companies disabled search features for primates globally, reducing app functionality for all users as a workaround.

Inferred additional harm: Other computer vision products likely exhibit similar performance disparities across different demographic groups due to unrepresentative training data.

Civil rights

Reported: Yes, the report describes accusations of racial bias and discrimination.

Directly caused: The misclassification of Black individuals as gorillas violated their right to equal treatment and freedom from racial discrimination.

Indirectly caused: N/A

Inferred additional harm: N/A

Psychological

Reported: Yes, the report describes emotional distress, dismay, and a loss of faith in technology.

Directly caused: The developer [PERSON_001] and his friend experienced significant offense and dismay upon discovering they were labeled as gorillas.

Indirectly caused: N/A

Inferred additional harm: Other Black users who experienced or learned of the classification likely felt alienated, degraded, and distressed by the perpetuation of racist tropes.

People affected

• Occurrences reported: 1
• People reportedly harmed: 2
• People reportedly exposed: 2

Potential causes

Management

• Prioritizing Speed Over Safety: Rushing public release without thoroughly checking for demographic biases.
• Risk Avoidance Over Resolution: Deciding to disable primate recognition to mitigate reputational risk.

Technology

• Inability to Distinguish Dark Skin: Models confused darker-skinned humans with primates due to representation gaps.
• Walling Off Malfunctioning Features: Disabling primate labels entirely instead of fixing algorithmic flaws.

Data Inputs

• Lack of Diverse Training Data: Insufficient photos of Black people in the training dataset caused bias.
• Troubling Data Collection Tactics: Targeting vulnerable groups to quickly gather dark-skinned facial scans.

Human Factors

• Overreliance on AI Technology: Society and developers put too much trust in unproven computer vision systems.

Process and Methods

• Inadequate Pre-release Testing: Not asking enough diverse employees to test the feature before public debut.
• Ineffective Model Verification: Failing to detect systemic classification errors during internal validation.

Regulatory Environment

• Absence of AI Bias Regulations: No regulatory standards forcing companies to audit training data for diversity.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, historical, and verified details about the Google Photos incident, including statements from former employees, spokespeople, and independent testing of competitor apps.
• Ambiguities identified: None of significance.
• Alternative interpretations: None.

The incident involves commercial photo-recognition software misclassifying Black individuals due to biased training data. While highlighting significant ethical concerns and algorithmic limitations in consumer technology, it poses negligible threat to national security, critical infrastructure, sovereignty, or strategic economic interests.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing, long-term challenge in AI development and data bias rather than an active, imminent national security crisis.
• Autonomy: Human-controlled. The AI model performs classification to assist users but operates within a consumer app environment with human intervention to disable features.
• Novelty: Established threat. Algorithmic bias and data underrepresentation are well-documented, established issues in machine learning and computer vision.

Impact by dimension

• Physical security: Negligible. The incident involves commercial consumer photo-tagging software with no connection to physical systems, critical infrastructure, or kinetic capabilities.
• Information security: Negligible. No intelligence compromise, classified data theft, or systematic state-sponsored information warfare operations are indicated in this incident.
• Sovereignty: Negligible. The issue is limited to commercial consumer software and does not impact government decision-making, elections, or state sovereignty.
• Economic security: Negligible. While it reveals limitations in commercial computer vision, it does not threaten strategic economic sectors or national technological security.
• Societal stability: Minor. Perpetuates harmful racial stereotypes and highlights algorithmic bias in consumer products, but lacks the scale or systemic deployment to threaten overall national societal stability.