NYPD Facial Recognition System Allegedly Produced Erroneous Match That Reportedly Resulted in Wrongful Detention of Trevis Williams

The NYPD's facial recognition system allegedly misidentified Trevis Williams as a suspect in a Union Square indecent exposure case. Despite reportedly notable physical differences and exculpatory phone data, Williams was arrested, jailed for more than two days, and charged. The case was later dismissed.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The facial recognition system failed to perform reliably under real-world conditions, generating a false match from grainy, low-quality surveillance footage.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Police investigators over-relied on the AI's 'possible match' output, using it to build a case and arrest the suspect without conducting basic corroborative checks.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was initiated by a false match generated by the facial recognition AI system during active law enforcement operations.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes the NYPD's use of facial recognition technology for law enforcement purposes, specifically to identify a criminal suspect from surveillance footage, which is classified as a high-risk application under the EU AI Act.

AI system and alleged parties

• AI system: NYPD facial recognition system
• AI purpose: Face Recognition; Identification
• Behaviour type: Assistant
• Alleged developer: Unknown facial recognition system developer
• Alleged deployer: NYPD, New York Police Department
• Alleged harmed parties: Trevis Williams, Law enforcement, Judicial system, Judicial integrity, General public of New York, General public

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report describes differential treatment where an innocent Black man was singled out and arrested due to a flawed algorithmic match.

Directly caused: The AI system flagged [PERSON_002] as a match despite major physical discrepancies, leading to him being targeted for a photo lineup.

Indirectly caused: He was arrested and jailed for two days based on a false identification, representing severe differential treatment compared to a non-suspect.

Inferred additional harm: Black individuals may be disproportionately subjected to wrongful investigations and arrests due to the systemic deployment of facial recognition systems on low-quality footage.

Civil rights

Reported: The report explicitly describes violations of civil rights resulting from the incident.

Indirectly caused: [PERSON_002]'s civil rights were violated when he was wrongfully arrested and detained in jail for more than two days based on a flawed facial recognition match.

Inferred additional harm: Similar civil rights violations likely occur to others, as the ACLU notes at least 10 known cases nationwide of wrongful arrests stemming from facial recognition.

Privacy

Reported: The report describes the NYPD's use of a mugshot database to run facial recognition searches.

Directly caused: The NYPD's facial recognition system processed [PERSON_002]'s mug shot from a previous dismissed case to match him to a crime scene.

Inferred additional harm: The continuous storage and algorithmic scanning of mug shots of individuals whose cases were dismissed poses ongoing privacy risks to many citizens.

Psychological

Reported: The report explicitly describes severe psychological distress experienced by the wrongfully arrested individual.

Indirectly caused: The wrongful arrest and detention caused [PERSON_002] to suffer from panic attacks, humiliation, anger, and severe anxiety over potentially being placed on a sex offender registry.

Inferred additional harm: The victim of the original crime also experienced a loss of tranquility and renewed fear upon learning that the wrong man had been arrested and the actual culprit remained at large.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Failure to Track System Errors: NYPD uses FRT thousands of times but does not track successes or failures.
• Overreliance on Algorithmic Leads: Management allowed arrests based on FRT leads without rigorous checks.

Technology

• FRT Performance Drop on Grainy Images: FRT accuracy drops significantly on grainy surveillance footage.
• Statistical Similarity Mismatch: Algorithm matched faces based on contours despite physical differences.

Data Inputs

• Low-Quality Surveillance Footage: Search used grainy, blurry images instead of controlled-condition photos.
• Mug Shot Database Inclusion: Active mug shot database included individuals with dismissed cases.

Human Factors

• Cognitive Bias in Photo Array: Victim selected the suspect due to suggestion and confirmation bias.
• Investigator Confirmation Bias: Detectives accepted the match without verifying physical discrepancies.
• Examiner Misidentification: Human examiner selected the wrong candidate from the FRT suggestions.

Process and Methods

• No Independent Corroboration Rule: NYPD does not require extra evidence before putting FRT match in lineup.
• Failure to Verify Alibi: Police did not check phone location data or contact Amazon to verify alibi.
• Inadequate Physical Verification: No process to reconcile physical differences like height (5'6 vs 6'2).

Regulatory Environment

• Lack of FRT Use Regulations: Absence of strict statutory limits on police facial recognition use.
• No Standardized Oversight: No external regulatory body auditing accuracy or usage of police FRT.

Information quality

• Classification confidence: High
• Reason for confidence: The report from the New York Times is highly detailed, providing specific names, dates, physical descriptions, and the sequence of events leading to the wrongful arrest. The role of the facial recognition system is explicitly described, along with the human decisions that compounded the error.
• Ambiguities identified: None of significance; the technical limitations of the system on grainy footage and the police's failure to verify the match are clearly laid out.
• Alternative interpretations: One could argue the primary failure was human police negligence rather than the AI system itself, though the AI's false match was the initiating event.

A local law enforcement incident where NYPD's overreliance on a facial recognition 'possible match' led to the wrongful arrest of an innocent man. While representing a serious civil rights concern, the incident has minor national security implications and is manageable within standard legal and judicial procedures.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: New York City Police Department

Threat characteristics

• Imminence: Long-term. The specific case is resolved, representing an ongoing strategic and policy concern regarding law enforcement use of AI rather than an active crisis.
• Autonomy: Human-controlled. The AI system only provided a possible match, and humans (NYPD detectives) made the final decision to arrest.
• Novelty: Established threat. Wrongful arrests stemming from facial recognition errors are an established issue, with multiple similar cases previously documented.

Impact by dimension

• Physical security: Negligible. The incident involved a local law enforcement investigation and did not affect physical systems, critical infrastructure, or kinetic military capabilities.
• Information security: Negligible. No intelligence capabilities, classified models, or national-level information operations were compromised or involved.
• Sovereignty: Negligible. Core national government functions, sovereignty, and electoral systems were not impacted by this local law enforcement action.
• Economic security: Negligible. The incident did not threaten financial systems, strategic industries, or national technological competitive advantage.
• Societal stability: Minor. The incident represents a minor national security impact, involving a localized civil rights violation and wrongful arrest of an individual due to facial recognition error, manageable through standard legal procedures.