Detroit Police Wrongfully Arrested Black Man Due To Faulty Facial Recognition

A Black man was wrongfully detained by the Detroit Police Department as a result of a false facial recognition (FRT) result.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The facial recognition software exhibited unequal performance across demographic groups, misidentifying Black faces at substantially higher rates than White faces, directly leading to the wrongful arrest of Black men.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI system failed to perform reliably, yielding a high rate of false matches (up to 96% according to the police chief).
• 5.1 Overreliance and unsafe use: Police officers over-relied on the AI's output, treating a mere lead as definitive proof of identity without conducting basic due diligence.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The wrongful arrests were triggered by an erroneous match generated by the deployed facial recognition AI, which was an unintended failure of the system.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used for law enforcement applications, specifically facial recognition, which is explicitly listed as a High Risk application under the EU AI Act.

AI system and alleged parties

• AI system: DataWorks Plus facial recognition software (DataWorks Plus)
• AI purpose: Face Recognition; Identification
• Behaviour type: Tool
• Alleged developer: DataWorks Plus
• Alleged deployer: Detroit Police Department
• Alleged harmed parties: Michael Oliver, Black people in Detroit

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Substantial
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Substantial
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes, the reports describe severe financial losses including job loss, car loss, and home loss, as well as a 12 million dollar lawsuit.

Indirectly caused: Michael Oliver lost his job, car, and home due to being detained and missing work for trial dates. He filed a lawsuit seeking at least 12 million dollars in damages.

Inferred additional harm: Robert Williams likely suffered financial losses from legal representation and missed work, though specific amounts are not detailed in the reports.

Differential treatment

Reported: Yes, the reports describe systemic differential treatment where the AI misidentifies Black and brown faces at substantially higher rates than White faces.

Directly caused: The AI system generated erroneous matches specifically targeting Black individuals due to demographic performance disparities.

Indirectly caused: Black residents in Detroit are disproportionately subjected to wrongful police investigations and arrests due to the deployment of this biased technology.

Inferred additional harm: Widespread systemic bias and differential treatment of Black residents by law enforcement using this software.

Civil rights

Reported: Yes, the reports describe violations of civil liberties and wrongful arrests.

Indirectly caused: Michael Oliver and Robert Williams had their civil rights violated through wrongful arrest and detention based on the erroneous AI match.

Inferred additional harm: Other individuals may have had their civil rights violated by wrongful detentions or investigations prompted by the software.

Privacy

Reported: Yes, the reports mention concerns about privacy and the use of databases containing mugshots and driver's license photos.

Indirectly caused: The police ran images through a database of state driver's license photos and mugshots without explicit consent, violating general privacy expectations.

Inferred additional harm: Mass surveillance of Detroit residents' faces without consent.

Psychological

Reported: Yes, the reports explicitly describe psychological harm including anxiety, fear, embarrassment, and shame.

Indirectly caused: Michael Oliver suffered severe anxiety, fear, embarrassment, and shame from being jailed for 2.5 days. Robert Williams experienced distress from being arrested in front of his family.

Inferred additional harm: It is highly likely that both victims and their families suffered long-term psychological distress and trauma from the wrongful arrests and subsequent legal battles.

Epistemic

Reported: Yes, the reports describe the generation of false identification matches that served as incorrect evidence.

Directly caused: The AI system generated false identification matches, creating incorrect investigative leads.

Indirectly caused: Police and prosecutors made decisions based on these false matches, leading to wrongful charges.

People affected

• Occurrences reported: 2
• People reportedly harmed: 2
• People reportedly exposed: 2

Potential causes

Management

• Prioritizing Tech Over Safety: Renewed controversial contracts despite known high error rates.
• Lack of Investigator Training: Management did not ensure investigators followed up on algorithmic leads.
• Insufficient Initial Protocols: No strict protocols were established for warrants using facial recognition.

Technology

• High Error Rate for Black Faces: The software has a substantial error rate among Black and brown individuals.
• Algorithmic Bias: Algorithms replicate racial bias programmed into them by developers.
• High General Misidentification Rate: Police chief acknowledged a 96 percent misidentification rate if used alone.

Data Inputs

• Low Quality Source Image: The system used a single blurry screen-grab from a cellphone video.
• Biased Database Photos: Matches were queried against mugshots and driver's license databases.

Human Factors

• Lack of Due Diligence: Detective failed to compare obvious physical features like tattoos.
• Confirmation Bias: The witness and detective uncritically accepted the algorithmic match.
• Lack of Witness Interview: Officer failed to interview a key student witness named in the video.

Process and Methods

• No Written Policy: The department had no written policy governing facial recognition at the time.
• Inadequate Photo Lineup Validation: An unverified algorithmic match was used directly to create a photo lineup.
• Lack of Prior Prosecutor Review: Supervisor did not review the facial recognition evidence before charging.

Regulatory Environment

• Absence of Local Ban: Detroit renewed its contract unlike other cities that banned the technology.
• Lack of Disclosure Requirements: Police were not required to share when matches were used, hiding errors.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the two wrongful arrests, including names, dates, the specific software used (DataWorks Plus), and the consequences. The role of the AI is explicitly stated and corroborated by police statements.
• Ambiguities identified: None significant; the mechanics of the misidentification and subsequent police actions are clear.
• Alternative interpretations: None; the causal link between the AI's false match and the wrongful arrests is clear.

The Detroit Police Department's reliance on biased facial recognition technology led to multiple wrongful arrests of Black men. While presenting no direct threat to national infrastructure, sovereignty, or economic security, the incident highlights substantial domestic concerns regarding civil rights violations, systemic algorithmic bias, and the erosion of societal trust in law enforcement.

• Overall national security impact: Substantial
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Detroit Police Department

Threat characteristics

• Imminence: Long-term. This represents an ongoing systemic concern regarding algorithmic bias in law enforcement rather than an active, immediate national security crisis.
• Autonomy: Human-supervised. The facial recognition software operated as a tool to generate leads, but police officers were responsible for verifying the outputs and executing arrests.
• Novelty: Evolved capability. This represents a significant and well-documented escalation of known demographic biases in facial recognition technology leading to concrete civil liberties harms.

Impact by dimension

• Physical security: Negligible. The incident did not involve any physical attacks, critical infrastructure compromise, or threats to human safety from kinetic systems.
• Information security: Negligible. No intelligence capabilities, classified models, or information warfare campaigns were involved or compromised in this incident.
• Sovereignty: Minor. While local law enforcement processes were compromised by bad data, this did not threaten national sovereignty, border control, or core federal government functions.
• Economic security: Negligible. The incident resulted in individual financial losses and local lawsuits, but it posed no threat to national economic stability or strategic technological advantages.
• Societal stability: Substantial. The wrongful arrests of Black men due to algorithmic bias and lack of due diligence represent a substantial violation of civil rights and contribute to the erosion of public trust in law enforcement.