Facial Recognition Misidentification at New World Westend in New Zealand

A facial recognition system at New World Westend supermarket misidentified a Māori woman as a known offender during its trial. The woman was wrongfully accused of trespassing and experienced public embarrassment, raising concerns about racial bias and the technology's accuracy. The supermarket acknowledged its error and apologized.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The incident represents unequal performance of the facial recognition system across demographic groups, as experts highlighted that the technology is less accurate for people of color, specifically Maori and Pasifika.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Supermarket staff over-relied on the AI's alert and failed to properly verify the match before confronting and accusing the customer.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was triggered by a false match generated by the deployed facial recognition AI system, leading to an unexpected and harmful outcome.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes a facial recognition system used for identification and surveillance in public/semi-public spaces, which has significant implications for fundamental rights and public interest.

AI system and alleged parties

• AI system: facial recognition technology
• AI purpose: Face Recognition; Identification
• Behaviour type: Assistant
• Alleged developer: Foodstuffs
• Alleged deployer: New World Westend, Foodstuffs
• Alleged harmed parties: Te Ani Solomon, Māori Community

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment of a Maori woman due to a false match by the facial recognition system.

Directly caused: The customer was singled out, accused of trespassing, and ordered to leave the store based on an incorrect match that experts attribute to demographic bias in the software.

Inferred additional harm: Other Maori and Pasifika shoppers are at a higher risk of being falsely accused and banned due to the system's lower accuracy for these demographic groups.

Privacy

Reported: The report explicitly describes privacy concerns regarding the highly invasive nature of facial recognition technology.

Directly caused: The customer's biometric data was scanned and compared against a database without her prior knowledge or explicit consent.

Inferred additional harm: Thousands of shoppers across the 25 trialling supermarkets are having their biometric data collected and processed, potentially violating their privacy expectations.

Psychological

Reported: The report explicitly describes psychological distress, humiliation, and fear experienced by the customer.

Directly caused: The customer experienced severe embarrassment, broke down in tears, and felt terrified and publicly humiliated during the 10-minute confrontation.

Indirectly caused: The incident ruined the customer's birthday and left her feeling unsafe and at risk of being falsely accused elsewhere.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Premature Trial Deployment: Deploying unproven, biased FRT across 25 supermarkets.
• Inadequate Staff Training: Staff training was insufficient to prevent critical human errors.

Technology

• Demographic Bias in FRT Software: FRT is less accurate for people of color and women of color.
• Offshore Training of AI Models: Software not trained on NZ population, reducing Maori accuracy.

Data Inputs

• Poor Database Image Quality: Comparison image was of low quality, leading to match error.
• Inadequate Database Verification: Database contained incorrect or poorly labeled suspect profiles.

Human Factors

• Staff Confirmation Bias: Staff assumed the system match was correct without critical review.
• Aggressive Staff Confrontation: Staff got in customer's face and demanded she leave immediately.
• Failure to Verify Identity: Staff ignored three forms of physical ID offered by the customer.

Process and Methods

• Flawed Verification Process: Staff failed to properly verify the match before acting on alert.
• Lack of Clear Escalation Steps: No process to de-escalate or double-check when customer contested.

Regulatory Environment

• Unproven Tech in Retail Spaces: FRT is not a proven tool to reduce retail crime in supermarkets.
• Lack of Strict FRT Regulations: Absence of strong legal limits on deploying invasive FRT trials.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide a clear, detailed account of the specific incident, including perspectives from the affected individual, the supermarket chain, and external experts. The role of the AI system (facial recognition) and the nature of the failure are explicitly documented.
• Alternative interpretations: The supermarket chain framed the incident primarily as a 'human error' in verification, whereas experts and the victim highlighted the systemic racial bias and technical limitations of the AI system itself.

A Maori woman was falsely accused of trespassing due to a false positive in a commercial facial recognition trial in New Zealand. The incident highlights ongoing concerns regarding demographic bias in biometric technology and privacy rights, but presents negligible direct threat to national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: New Zealand
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Reflects an ongoing strategic or ethical concern regarding biometric bias rather than an active national security crisis.
• Autonomy: Human-supervised. The facial recognition system flagged the match autonomously, but protocol required human verification before action was taken.
• Novelty: Established threat. Demographic bias in facial recognition software and false-positive mismatches are well-documented, established technological issues.

Impact by dimension

• Physical security: Negligible. The incident occurred in a retail supermarket and involved no physical threats, kinetic attacks, or critical infrastructure damage.
• Information security: Negligible. No information warfare, systematic disinformation, or compromise of state intelligence capabilities occurred.
• Sovereignty: Negligible. The incident involved a private retail trial and did not disrupt government operations, electoral systems, or state sovereignty.
• Economic security: Negligible. The incident represents a local commercial trial failure with no broader impact on national economic stability or strategic technological sectors.
• Societal stability: Minor. Highlights issues of algorithmic bias and privacy concerns for indigenous populations, but the immediate impact is limited to a localized commercial trial.