Passport checker Detects Asian man's Eyes as Closed

New Zealand passport robot reader rejects the application of an applicant with Asian descent and says his eyes are closed.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The facial recognition software performed poorly on an individual of Asian descent due to his epicanthal folds, demonstrating unequal accuracy and effectiveness based on demographic characteristics.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the automated passport photo checker's failure to correctly process the user's photo, which was an unexpected outcome of its deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk (Level 2): The system is a biometric verification application used by a government department for passport control, which has significant implications for public interest and fundamental rights.

AI system and alleged parties

• AI system: New Zealand Department of Internal Affairs online passport photo checker
• AI purpose: Face Recognition; Image Verification
• Behaviour type: Tool
• Alleged developer: New Zealand
• Alleged deployer: New Zealand
• Alleged harmed parties: Asian People

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment where an Asian man's photo was rejected due to facial features characteristic of his ethnicity.

Directly caused: The automated system rejected Richard Lee's photo because it misidentified his open eyes as closed, a failure linked to his epicanthal folds.

Indirectly caused: The incident led to public allegations of algorithmic bias and systemic insensitivity toward people of Asian descent.

Inferred additional harm: It is highly likely that other individuals with similar facial features experienced similar rejections and administrative delays when using this system.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• High Accepted Rejection Rate: Management accepted a high 20 percent rejection rate as normal for the system.
• Reliance on Generic Errors: Management deployed a system that used generic errors for complex failures.

Technology

• Flawed Eye Detection Algorithm: The software interpreted epicanthal folds as being closed.
• Sensitivity to Eye Shadows: Uneven facial lighting and eye shadows caused the system to misread the eyes.
• Generic Error Message System: System sent a generic 'eyes closed' error instead of explaining actual issues.

Data Inputs

• Uneven Facial Lighting: The uploaded photograph had shadows and uneven lighting on the face.
• Biased Algorithm Design: System was not optimized for diverse facial features like epicanthal folds.

Human Factors

• User Uploads Shadowed Photo: The applicant uploaded an initial photo with uneven lighting and shadows.

Process and Methods

• Inflexible Automated Rejection: The system automatically rejected the photo without immediate human review.
• No Immediate Human Escalation: No automatic routing to staff after multiple failed online submissions.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the incident, including statements from the affected individual and the government department. The technical cause of the failure (lighting/shadows vs. algorithmic bias on epicanthal folds) is discussed clearly from both perspectives.
• Ambiguities identified: There is some ambiguity regarding whether the system's failure was purely due to lighting/shadows as the department claimed, or primarily due to algorithmic bias against Asian facial features as critics suggested.
• Alternative interpretations: The incident could be interpreted purely as a routine technical glitch related to lighting rather than systemic algorithmic bias.

An automated passport photo checker in New Zealand rejected an Asian man's photo due to facial recognition bias, misidentifying his eyes as closed. While highlighting ongoing challenges with demographic disparities in AI, the incident has negligible national security implications and was resolved administratively.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: New Zealand
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Concerns about algorithmic bias represent ongoing, long-term technological and societal challenges rather than an active crisis.
• Autonomy: Human-supervised. The automated checker made the initial rejection decision autonomously, but human staff were available to review and process the application.
• Novelty: Established threat. Algorithmic bias and demographic disparities in facial recognition performance are well-documented and established issues.

Impact by dimension

• Physical security: Negligible. No physical threat, kinetic attack, or critical infrastructure disruption occurred.
• Information security: Negligible. No intelligence compromise, espionage, or disinformation campaigns were associated with this administrative incident.
• Sovereignty: Minor. The incident involved an automated government passport system, causing minor administrative delays, but did not threaten state authority or core functions.
• Economic security: Negligible. No strategic technology theft, financial system attack, or economic destabilization occurred.
• Societal stability: Minor. The system demonstrated algorithmic bias against individuals with Asian facial features, causing public discussion on discrimination but no widespread instability.