Wrongful Attempted Arrest for Apple Store Thefts Due to NYPD’s Facial Misidentification

New York Police Department (NYPD)’s facial recognition system falsely connected a Black teenager to a series of thefts at Apple stores, which resulted in his wrongful attempted arrest.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The facial recognition system failed to perform robustly and accurately, misidentifying the plaintiff as the thief despite significant physical differences.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The misidentification of two Black individuals who looked nothing alike highlights potential racial bias and misrepresentation in facial recognition performance.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The harm resulted from the post-deployment operation of a facial recognition system that unintentionally misidentified the plaintiff due to incorrect database associations.

EU AI Act risk tier

• Risk tier: 2 High Risk

Risk Level 2: High Risk. The system is a facial recognition application used for law enforcement and security purposes, which is explicitly classified as high-risk under the EU AI Act.

AI system and alleged parties

• AI system: Apple security technology, NYPD facial recognition system (Apple)
• AI purpose: Face Recognition; Identification
• Behaviour type: Assistant
• Alleged developer: unknown
• Alleged deployer: New York Police Department
• Alleged harmed parties: Ousmane Bah, NYC Black young people, NYC Black people

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Minor
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes, the report describes a lawsuit seeking $1 billion in damages, and mentions academic and reputational harm.

Indirectly caused: Bah suffered unquantified financial losses related to legal defense fees, travel to out-of-state courts, and academic setbacks from missing school and exams.

Differential treatment

Reported: Yes, the report describes the plaintiff being repeatedly targeted and prosecuted based on incorrect database records.

Directly caused: The facial recognition system matched the thief's face to Bah's identity, leading to him being treated as a suspect.

Indirectly caused: Bah was singled out for arrest and prosecution across multiple states while the actual thief was initially not prosecuted under his own name.

Civil rights

Reported: Yes, the report explicitly describes wrongful arrest and false prosecution.

Indirectly caused: Bah's civil liberties were violated when he was wrongfully arrested at his home at 4 AM and subjected to malicious prosecution across multiple states.

Privacy

Reported: Yes, the report mentions a class action lawsuit alleging illegal collection and storage of face scans.

Indirectly caused: Bah's personal information was stored and linked to biometric surveillance data in security databases without his authorization.

Psychological

Reported: Yes, the report explicitly describes severe stress, hardship, and emotional distress.

Indirectly caused: Ousmane Bah suffered severe stress, hardship, and emotional distress due to being wrongfully arrested at his home at 4 AM and facing multiple false criminal charges.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Negligent Security Oversight: Apple and SIS failed to audit incorrect identity links in their databases.
• Failure to Rectify Known Errors: Management continued using flawed data after mismatches were identified.

Technology

• Facial Recognition Misidentification: System matched surveillance footage of impostor to Bah's incorrect record.
• NYPD Database Search Confusion: Facial recognition search returned both the real thief and the victim.

Data Inputs

• Flawed Identity Association: Impostor's photo was linked with Bah's name in the security database.
• Non-Photo ID Vulnerability: Temporary learner's permit lacked a photo, allowing easy identity theft.

Human Factors

• Failure to Verify Physical Matches: Staff ignored obvious height and physical differences between thief and ID.
• Overreliance on System Alerts: Security and police accepted automated matches without manual verification.

Process and Methods

• Flawed BOLO Creation Process: BOLO notices were generated linking victim's name to the thief's photo.
• Negligent Evidence Retention: Apple and SIS failed to retain or deleted crucial store surveillance videos.

Regulatory Environment

• Lack of Biometric Regulations: No strict legal standards governed private facial recognition database use.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide detailed, consistent accounts of the legal proceedings, the timeline of events, and the specific role of the facial recognition database in propagating the misidentification.
• Ambiguities identified: There is some ambiguity regarding the exact extent of Apple's direct use of facial recognition versus that of its contractor SIS.
• Alternative interpretations: The incident could be viewed primarily as a case of human negligence and identity theft, with the AI system acting merely as a tool that scaled the error.

An 18-year-old student was wrongfully arrested due to a facial recognition misidentification propagated across private security and police databases. Although the incident highlights significant civil liberties and biometric privacy concerns, it has negligible implications for national security, critical infrastructure, or state sovereignty.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Individual

Threat characteristics

• Imminence: Long-term. The specific case is resolved, representing an ongoing strategic concern regarding domestic biometric data regulation rather than an active crisis.
• Autonomy: Human-supervised. The facial recognition system flagged the match, but human security and law enforcement personnel made the final decision to arrest and prosecute.
• Novelty: Established threat. Facial recognition misidentification and database correlation errors are known technical limitations that have occurred in previous instances.

Impact by dimension

• Physical security: Negligible. The incident involved retail thefts and wrongful arrests, presenting no threat to physical systems, critical infrastructure, or human safety at a national scale.
• Information security: Negligible. No information warfare, state-sponsored intelligence compromise, or systematic disinformation campaigns were involved in this localized case.
• Sovereignty: Negligible. While local police were involved in the arrests, the incident did not compromise core national government decision-making or state sovereignty.
• Economic security: Negligible. The incident involved retail shoplifting and private security databases, representing no threat to strategic industries or national economic security.
• Societal stability: Minor. The wrongful arrest violated the civil liberties of an individual, highlighting localized risks of biometric surveillance, but did not threaten national social cohesion.