Images of Black People Labeled as Gorillas

Google Photos image processing software mistakenly labelled a black couple as "gorillas."

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The AI system exhibited lower accuracy and performance for darker-skinned individuals due to a lack of diverse training data, leading to offensive misclassifications.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The system's output associated Black individuals with a deeply offensive racist trope, representing them as primates.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The misclassification was an unexpected error made by the Google Photos image recognition AI after it was deployed to the public.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The system is a consumer photo organization and search application, which poses low or negligible risk to safety or fundamental rights under the EU AI Act.

AI system and alleged parties

• AI system: Google Photos image recognition software (Google)
• AI purpose: Image Tagging; Image Search
• Behaviour type: Assistant
• Alleged developer: Google
• Alleged deployer: Google
• Alleged harmed parties: Black people

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes the generation of offensive and racist labels by the AI system.

Directly caused: The Google Photos app generated and applied the highly offensive label 'gorillas' to photos of Black individuals.

Differential treatment

Reported: The report explicitly describes differential treatment where the AI system performed poorly specifically for darker-skinned individuals.

Directly caused: The image recognition software failed to accurately recognize Black faces, misclassifying them as animals, while performing accurately for white faces.

Inferred additional harm: The systemic bias in training data likely resulted in ongoing poorer service quality and misclassifications for users of color globally.

Psychological

Reported: The report explicitly describes psychological harm in the form of dismay, offense, and outrage experienced by the affected user.

Directly caused: Jacky Alcine and his friend experienced significant distress, dismay, and offense upon seeing themselves labeled as gorillas.

Inferred additional harm: It is likely that other Black users who realized the system's limitations or heard of the incident experienced racial trauma and distress, though this is not quantified.

People affected

• Occurrences reported: 1
• People reportedly harmed: 2
• People reportedly exposed: 2

Potential causes

Management

• Prioritization of Speed to Market: Releasing cutting-edge tech quickly with the plan to fix bugs post-launch.
• Lack of Diversity Initiatives: Slow progress in recruiting and retaining Black and Latino technical staff.

Technology

• Nascent Computer Vision Models: Algorithms rely on basic patterns and lack human common sense.
• Black Box Nature of ML: Creators struggle to understand how models make classification decisions.
• Inadequate Contrast Processing: Image sensors and software struggle with dark skin tones and low lighting.

Data Inputs

• Lack of Diverse Training Data: Training datasets lacked sufficient images of dark-skinned individuals.
• Biased Dataset Selection: Bias in collected images led to poor representation of people of color.

Human Factors

• Lack of Diversity in Tech Teams: Homogeneous engineering teams failed to anticipate or test for diverse users.
• Overreliance on Automation: Assuming automated tagging would perform flawlessly without human oversight.

Process and Methods

• Insufficient QA Testing: The app was not tested on a sufficiently diverse focus group before release.
• Ineffective Workaround Fixes: Blocking primate labels rather than retraining the underlying neural network.
• Lack of Pre-release Employee Testing: Not enough employees were asked to test the feature before its public debut.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the 2015 Google Photos incident, including direct quotes from the affected user, Google executives, and subsequent technical investigations by Wired. The facts of the misclassification and Google's subsequent workaround are well-documented across multiple reputable sources.

An unintentional post-deployment misclassification by Google Photos in 2015 labeled Black individuals as 'gorillas'. While causing significant public outcry and highlighting critical issues of racial bias in AI training datasets, the incident has negligible national security implications, representing a minor societal impact managed through corporate software adjustments.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents a long-term strategic concern regarding algorithmic bias and dataset diversity rather than an active, imminent national security crisis.
• Autonomy: Full autonomy. The image recognition system automatically classified and tagged photos without human intervention or pre-moderation before displaying them to users.
• Novelty: First-of-its-kind. This was one of the earliest and most prominent public examples of severe racial bias in commercial computer vision models, establishing a major precedent for AI ethics.

Impact by dimension

• Physical security: Negligible. The incident was a software classification error in a consumer photo application and posed no threat to physical systems, critical infrastructure, or human safety.
• Information security: Negligible. No intelligence capabilities, classified information, or state-sponsored information warfare operations were involved or compromised.
• Sovereignty: Negligible. The incident did not impact state authority, government decision-making, electoral systems, or any core government operations.
• Economic security: Negligible. While representing a reputational issue for a major technology firm, the incident did not threaten national economic stability, strategic industries, or technological competitive advantage.
• Societal stability: Minor. The incident caused significant public offense and highlighted systemic racial bias in AI training data, but it did not lead to large-scale civil unrest, state-level human rights abuses, or national instability.