Emotion Detection Models Showed Disparate Performance along Racial Lines

Emotion detection tools by Face++ and Microsoft's Face API allegedly scored smiling or defaulted ambiguous facial photos for Black faces as negative emotion more often than for white faces.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The AI systems perform unequally across demographic groups, exhibiting significantly lower accuracy and higher rates of negative emotion misclassification for Black faces compared to white faces.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The systems unfairly represent Black individuals by associating their facial expressions with negative emotions like anger and contempt, reinforcing harmful racial stereotypes.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The racial bias is an unexpected and unintentional outcome of the AI systems' emotion classification models, which manifested after the models were trained and deployed.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes facial recognition systems being used in recruitment ('hiring decisions') and public safety ('scan the faces in crowds to identify threats'), which are explicitly classified as high-risk applications under the EU AI Act.

AI system and alleged parties

• AI system: Face API, Face++ (Face++, Microsoft)
• AI purpose: Face Recognition; Threat Detection
• Behaviour type: Tool
• Alleged developer: Microsoft, Megvii
• Alleged deployer: Microsoft, Megvii
• Alleged harmed parties: Black people

Harm severity

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Substantial
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment of individuals based on race caused directly by the AI systems.

Directly caused: The AI systems consistently assigned more negative emotional scores (anger and contempt) to Black faces compared to white faces, even when both groups exhibited similar smiles.

Indirectly caused: The deployment of these systems in hiring and public safety scanning leads to systemic disparate impact, where Black individuals are unfairly judged as angry or threatening.

Inferred additional harm: Widespread discriminatory outcomes in employment and law enforcement profiling against Black individuals due to the integration of biased algorithms.

Civil rights

Reported: The report explicitly mentions that the technology can perpetrate and exacerbate existing power dynamics, leading to disparate impact across racial groups.

Directly caused: N/A

Indirectly caused: The use of biased facial recognition in hiring and public surveillance threatens civil rights, specifically equal employment opportunity and protection against discriminatory profiling.

Inferred additional harm: Systemic civil rights violations if law enforcement or employers rely on these biased tools to make critical decisions affecting Black individuals.

People affected

• Occurrences reported: 1
• People reportedly harmed: 400
• People reportedly exposed: 400

Potential causes

Management

• Ignoring Disparate Impact: Failure to prevent technology from exacerbating existing power dynamics.
• Invisibility of AI Decisions: Keeping system operations and biases hidden from the people affected.

Technology

• Biased Classification Algorithms: Algorithms interpret black faces as angrier or more contemptuous.
• Formalization of Stereotypes: Software formalizes preexisting human stereotypes into automatic algorithms.

Data Inputs

• Biased Training Datasets: Training data likely reflects human biases in emotion perception.

Human Factors

• Preexisting Human Stereotypes: People perceive black men as more threatening, influencing tech design.
• Misplaced Trust in Objectivity: Belief that algorithms are naturally objective and free from human bias.

Process and Methods

• Inadequate Demographic Audits: Lack of testing to ensure similar emotional assessment across racial groups.

Regulatory Environment

• Lack of Public Accountability: No societal oversight to ensure fairness for groups affected by AI.

Information quality

• Classification confidence: High
• Reason for confidence: The analysis is based on a structured scientific study with clear quantitative findings comparing Face++ and Microsoft Face API. The methodology, dataset (400 NBA player photos), and specific bias metrics are clearly detailed in the reports.
• Ambiguities identified: None of significance; the study's parameters and results are clearly laid out.
• Alternative interpretations: None.

A scientific study revealed racial bias in facial recognition and emotion analysis software from Face++ and Microsoft, consistently misinterpreting Black faces as having more negative emotions. While posing concerns for civil rights and fair treatment in hiring and law enforcement, the incident has negligible direct national security implications and represents an established algorithmic challenge.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Multiple nations
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This represents an ongoing, systemic algorithmic bias issue rather than an active, time-sensitive national security crisis.
• Autonomy: Human-controlled. The AI systems function as tools to assist human decision-makers in hiring or public safety, rather than executing autonomous actions.
• Novelty: Established threat. Racial bias in facial recognition and computer vision models is a well-documented and established issue in the AI field.

Impact by dimension

• Physical security: Negligible. The incident involves software bias in emotion recognition and does not present physical threats, kinetic attacks, or critical infrastructure disruption.
• Information security: Negligible. No evidence of active information warfare, intelligence compromise, or hostile state-sponsored disinformation campaigns.
• Sovereignty: Negligible. No disruption to state authority, electoral systems, or core government decision-making processes is reported.
• Economic security: Negligible. While the bias could affect individual employment opportunities, it does not pose a systemic threat to national economic stability or strategic technological security.
• Societal stability: Minor. Algorithmic bias in emotion detection could lead to discriminatory outcomes in law enforcement and hiring, but the current impact is manageable within standard legal frameworks.