Uber Locked Indian Drivers out of Accounts Allegedly Due to Facial Recognition Fails

Uber drivers in India reported being locked out of their accounts allegedly due to Real-Time ID Check's facial recognition failing to recognize appearance changes or faces in low lighting conditions.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The facial recognition system exhibits unequal performance and higher error rates for Indian faces and minority ethnic groups, leading to disproportionate account lockouts.

Additional risk subdomains

• 6.2 Increased inequality and decline in employment quality: The automated lockouts degrade gig work quality and cause sudden loss of livelihood for vulnerable platform workers.
• 7.3 Lack of capability or robustness: The underlying AI system fails to perform robustly under common real-world conditions like low lighting or minor appearance changes.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The lockouts were caused by the unexpected failure of the deployed facial recognition AI system to accurately identify drivers under varying real-world conditions.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used for employment and worker management, specifically controlling access to work and livelihood, and utilizes biometric facial recognition, both of which are classified as high-risk applications under the EU AI Act.

AI system and alleged parties

• AI system: Microsoft Cognitive Services (Microsoft)
• AI purpose: Face Recognition; Workforce Monitoring and Evaluation
• Behaviour type: Autonomous
• Alleged developer: Uber, Azure Cognitive Services
• Alleged deployer: Uber
• Alleged harmed parties: Uber drivers in India

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment, noting that the facial recognition software performs worse on Indian faces and ethnic minority groups.

Directly caused: The AI system failed to recognize Indian faces at a higher rate (over 3% error rate in audits) and disproportionately misidentified minority ethnic drivers in the UK.

Inferred additional harm: Systemic bias in the training data of commercial facial recognition APIs likely leads to widespread, ongoing differential treatment of non-white gig workers globally.

Civil rights

Reported: The report explicitly mentions allegations of a 'racist algorithm' and lawsuits in the UK regarding wrongful termination and lack of algorithmic transparency.

Directly caused: Drivers' basic labor rights and right to work were compromised by automated account terminations without transparent recourse.

Inferred additional harm: The lack of regulatory protections in India likely leaves hundreds of thousands of gig workers vulnerable to systemic violations of their economic and civil rights.

Psychological

Reported: The report explicitly describes psychological distress, frustration, and a sense of defeat experienced by locked-out drivers.

Indirectly caused: Drivers like Adnan Taqi expressed deep frustration and a sense of defeat due to being locked out of their accounts and facing an unhelpful support system.

Inferred additional harm: It is highly likely that hundreds of other locked-out drivers experienced severe anxiety, stress, and mental distress due to the sudden and opaque loss of their primary source of income.

People affected

• Occurrences reported: 1
• People reportedly harmed: 125
• People reportedly exposed: 600000

Potential causes

Management

• Risk Shifted to Workers: Uber shifts the operational and financial risk of AI failures onto drivers.
• Prioritizing Security over Usability: Implementing strict fraud checks without safeguards for false matches.
• Lack of Transparency in Bans: Management fails to provide clear explanations for account deactivations.

Technology

• Brittle Facial Recognition Software: Microsoft Face API struggles to recognize faces with facial hair or shaved heads.
• Poor Edge Detection in Low Light: The software fails to detect facial edges in low lighting or low contrast.
• Failure on Diverse Demographics: The facial recognition tool has higher error rates on Indian and minority faces.

Data Inputs

• Inadequate Demographic Training Data: The underlying AI model was not sufficiently trained on diverse Indian faces.
• Low-Quality Selfie Submissions: Scratched phone cameras and low-budget smartphones produce low-quality inputs.
• Variable Lighting Conditions: Selfies taken at dusk or night lack the contrast required for verification.

Human Factors

• Natural Changes in Appearance: Drivers shaving heads, growing facial hair, or changing haircuts triggers fails.
• Driver Desperation and Fatigue: Drivers working extremely long hours face high pressure to bypass lockout bugs.
• Lack of Driver Tech Literacy: Drivers struggle to understand how to optimize lighting and angles for the AI.

Process and Methods

• Flawed Manual Review Process: Human reviewers fail to timely override false negatives, causing long lockouts.
• Ineffective Grievance Redress: Support systems are tedious, slow, and unhelpful for locked-out drivers.
• Frequent Verification Prompts: Insisting on real-time ID checks before rides increases the chance of failure.

Regulatory Environment

• Lack of Gig Worker Protections: India lacks policy frameworks to protect gig workers from algorithmic errors.
• Absence of Transparency Mandates: No regulations force platforms to disclose why an AI model rejected a driver.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly detailed, drawing on a structured survey by MIT Technology Review, academic audits of the Microsoft Face API, specific driver case studies, and statements from both driver unions and Uber.
• Ambiguities identified: There is a conflict between Uber's claim that no deactivations occur without manual review by at least two humans, and the drivers' experiences of immediate, automated lockouts with no effective human recourse.
• Alternative interpretations: Uber suggests some lockouts are due to community guideline violations rather than facial recognition failures, presenting an alternative explanation for specific cases like Neradi Srikanth.

Widespread account lockouts of Uber drivers in India and the UK due to facial recognition failures in Microsoft Azure's Face API, which exhibits higher error rates on Indian and minority ethnic faces. This algorithmic bias caused immediate financial harm and highlighted labor rights concerns, but carries minor national security implications.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: India, United Kingdom
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding algorithmic management and bias rather than an active crisis requiring urgent national response.
• Autonomy: Human-supervised. Uber claims human reviewers manually check flagged photos before deactivation, though drivers dispute the effectiveness of this oversight.
• Novelty: Established threat. Algorithmic bias and facial recognition failures on darker skin tones are well-documented, established issues.

Impact by dimension

• Physical security: Negligible. No physical threats, kinetic attacks, or critical infrastructure compromise reported.
• Information security: Negligible. No intelligence compromise, classified model theft, or state-sponsored information warfare detected.
• Sovereignty: Negligible. Commercial platform operational failure with no disruption to state authority or government functions.
• Economic security: Minor. Caused sudden financial loss and vehicle loan payment issues for hundreds of gig workers, but has limited impact on national economic or technological security.
• Societal stability: Minor. Algorithmic bias and unequal performance on Indian and minority ethnic faces led to loss of livelihood and labor rights concerns, but did not cause widespread civil unrest.