Facial Recognition Pilot in Bahia Reportedly Targeted Black and Poor People

Facial recognition deployed in a pilot project by the local government of Bahia despite having minimal hit rate reportedly targeted Black and poor people disproportionately.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The facial recognition system exhibits unequal performance across demographic groups, specifically suffering from higher error rates and false matches when processing Black faces compared to white faces.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The system suffers from a severe lack of robustness and accuracy, as evidenced by the report that only 3.6 percent of generated alerts led to actual arrest warrants.
• 5.1 Overreliance and unsafe use: Police officers over-relied on the AI's automated alert, tracking an innocent citizen for 22 kilometers and conducting an invasive stop based solely on the system's suggestion.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The wrongful stop and tracking of PERSON_001 was caused by an erroneous match generated by the deployed facial recognition AI system, which was an unintended outcome of its operation.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is a law enforcement application utilizing facial recognition in public spaces, which is explicitly classified as High Risk under the EU AI Act.

AI system and alleged parties

• AI system: facial recognition system (Huawei, Iecisa)
• AI purpose: Face Recognition; Smart Surveillance
• Behaviour type: Assistant
• Alleged developer: Huawei
• Alleged deployer: the government in Bahia, Bahia's Secretary of Public Security
• Alleged harmed parties: Black people in Brazil, Black people in Bahia

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Substantial
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Substantial
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes systemic differential treatment of Black individuals caused directly and indirectly by the incident.

Directly caused: PERSON_001, a Black man, was disproportionately targeted, tracked, and stopped by police due to algorithmic bias that misidentifies Black faces at higher rates.

Indirectly caused: The deployment of the biased system in Bahia, which has a majority Black population, systematically subjects Black citizens to higher rates of wrongful surveillance and police stops.

Inferred additional harm: The system likely reinforces existing racial profiling and systemic discrimination within the state's law enforcement practices, disproportionately affecting thousands of Black residents.

Civil rights

Reported: The report explicitly describes violations of civil rights, including freedom of movement and protection against arbitrary surveillance.

Directly caused: PERSON_001's civil rights were violated when he was tracked across 15 metro stations and subjected to an arbitrary police stop based on a false AI match.

Inferred additional harm: The widespread deployment of real-time public facial recognition threatens the fundamental civil liberties and privacy rights of millions of citizens in Bahia.

Privacy

Reported: The report explicitly describes privacy violations, including tracking individuals' movements and using social media photos without consent.

Directly caused: PERSON_001's public movements were tracked and recorded across 22 kilometers without his consent or prior suspicion of a crime.

Indirectly caused: The system aggregates and archives citizens' facial images, locations, and timestamps, and utilizes public social media photos for police investigations.

Inferred additional harm: The continuous, automated tracking of millions of citizens in public spaces constitutes a massive, ongoing violation of collective privacy rights.

Psychological

Reported: The report explicitly describes psychological harm, specifically fear, anxiety, and paranoia, experienced by the wrongfully stopped individual.

Directly caused: PERSON_001 experienced significant fear and anxiety after being tracked and stopped, leading him to contemplate changing his physical appearance (hair) to avoid future police harassment.

Inferred additional harm: It is highly likely that many of the other hundreds of individuals wrongfully flagged and stopped by police due to system errors experienced similar psychological distress, humiliation, and anxiety.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 903

Potential causes

Management

• Prioritizing Tech over Safety: State promoted facial recognition as a showcase despite low accuracy rates.
• Lack of System Accountability: Management expanded the system without resolving high rate of false alerts.

Technology

• Algorithmic Racial Bias: AI algorithms have higher error rates and false matches for Black individuals.
• Inaccurate Facial Matching: System triggered a false alert at 90 percent similarity for an innocent person.
• Inadequate Algorithm Training: Algorithm not trained on local demographics to prevent high false matches.

Data Inputs

• Flawed Arrest Databases: Warrant databases contain outdated, incorrect, or racially biased records.
• Unregulated Social Media Use: Police use public social media photos to feed the comparison database.
• Biased Training Datasets: Datasets lack diverse representation, leading to racial misidentification.

Human Factors

• Confirmation Bias in Review: Human operators validated the flawed 90 percent match instead of catching error.
• Police Reliance on AI Alerts: Officers trusted the automated alert, leading to tracking and stop of citizen.

Process and Methods

• Unrecorded False Positives: False identifications resolved on site are not recorded, hiding system errors.
• Inadequate Verification Protocol: Lack of a rigorous method to verify system alerts before initiating police stop.

Regulatory Environment

• Lack of Surveillance Regulation: No clear legal framework limits facial recognition use in public spaces.
• Absence of Data Privacy Controls: No strict controls on integrating private cameras or social media data.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides a detailed first-hand account of a specific victim (PERSON_001), statistical data on system error rates (3.6 percent accuracy), and expert analysis on algorithmic bias. The roles of the AI developers (Iecisa, Huawei) and the government are clearly defined, leaving little ambiguity about the nature of the failure.
• Ambiguities identified: The exact technical parameters of the training data used by Iecisa and Huawei are not fully disclosed, though experts infer a lack of diverse training data.
• Alternative interpretations: None. The incident is clearly an AI-driven false positive resulting in a wrongful police stop.

The deployment of a biased facial recognition system in Bahia, Brazil, highlights significant domestic societal stability and human rights concerns. The system disproportionately targets Black citizens due to high false-positive rates, leading to wrongful police stops and civil rights violations. While the threat to physical infrastructure is negligible, the state-wide expansion of foreign-sourced surveillance technology poses long-term concerns for societal trust and governance.

• Overall national security impact: Substantial
• Scope: Single nation
• Primary target: Brazil
• Alleged perpetrator: State Government of Bahia

Threat characteristics

• Imminence: Long-term. Represents an ongoing systemic issue and strategic concern regarding public surveillance rather than an immediate crisis.
• Autonomy: Human-supervised. The AI system automatically identifies and tracks targets, but human police officers make the final decision to stop and interrogate individuals.
• Novelty: Established threat. Racial bias and false positives in biometric facial recognition systems are well-documented and recurring issues globally.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic attacks, or critical infrastructure compromises were reported in connection with this incident.
• Information security: Negligible. The incident does not involve information warfare, intelligence compromise, or systematic foreign disinformation campaigns.
• Sovereignty: Minor. Involvement of foreign technology providers like Huawei in domestic law enforcement infrastructure poses minor long-term dependency and sovereignty risks.
• Economic security: Negligible. No significant threats to national economic stability, financial systems, or critical technology theft were identified.
• Societal stability: Substantial. Widespread deployment of biased facial recognition technology violates civil liberties, disproportionately targets Black citizens, and erodes trust in public safety institutions.