Facial Recognition System in Buenos Aires Triggers Police Checks Based on False Matches

Buenos Aires's facial recognition system mistakenly flagged innocent people as criminals, leading to wrongful stops and detentions. Judicial investigations indicate the technology may have been misused for unauthorized surveillance and data collection. Despite privacy risks, the system has been used widely without full disclosure of standards or safeguards,

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The system failed to perform reliably, generating at least 140 false positive database errors that led to the wrongful arrest and detention of innocent citizens.

Additional risk subdomains

• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: The system was allegedly misused to compile a Big Data database and access the biometric data of journalists without authorization.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was primarily caused by the AI facial recognition system generating false positive matches after its deployment, which was an unintentional and unexpected outcome of its public security goal.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes a facial recognition system used in public spaces for law enforcement, which fits the definition of 'Law enforcement applications, including facial recognition' under High Risk.

AI system and alleged parties

• AI system: facial recognition system
• AI purpose: Face Recognition; Smart Surveillance
• Behaviour type: Autonomous
• Alleged developer: Government of Argentina
• Alleged deployer: Government of Buenos Aires, Government of Argentina, Argentinean Ministry of Security
• Alleged harmed parties: Guillermo Ibarrola, Buenos Aires residents, Argentinean citizens

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Minor, indirect Minor
• Democracy: direct Minor, indirect Minor
• Privacy: direct Minor, indirect Severe
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment of individuals through wrongful police checks and arrests.

Directly caused: The system misidentified 140 innocent individuals, subjecting them to police checks and wrongful detention while actual offenders remained free.

Inferred additional harm: Given global patterns in facial recognition, the system may have disproportionately misidentified individuals based on race or demographic characteristics, though specific local data is not provided.

Civil rights

Reported: The report explicitly describes violations of human and civil rights caused directly and indirectly by the incident.

Directly caused: At least 140 innocent citizens had their civil liberties violated through wrongful police stops, searches, and detentions of up to six nights.

Indirectly caused: The potential misuse of the system for unauthorized surveillance violates the fundamental right to privacy and freedom of movement.

Inferred additional harm: The ongoing deployment of unregulated facial recognition threatens the broader civil liberties of the entire population of Buenos Aires.

Democracy

Reported: The report explicitly describes potential erosion of democratic norms through unauthorized surveillance.

Directly caused: The Ministry of Security requested the biometric data of journalists investigating the system, representing a potential abuse of power and intimidation.

Indirectly caused: The lack of transparency, control mechanisms, and oversight regarding the system's algorithms and data storage undermines public trust in law enforcement and democratic institutions.

Inferred additional harm: Unregulated surveillance systems can be used by authorities to monitor political opponents, activists, and journalists, chilling free speech and democratic participation.

Privacy

Reported: The report explicitly describes privacy violations caused directly and indirectly by the incident.

Directly caused: The biometric data of the investigating journalists was requested and accessed by the Ministry of Security without their consent or clear justification.

Indirectly caused: The system harvested and stored biometric data of citizens in public spaces without clear standards, control mechanisms, or transparency regarding data retention.

Inferred additional harm: The biometric privacy of millions of Buenos Aires residents was likely compromised through continuous, unregulated public surveillance and data harvesting.

Psychological

Reported: The report explicitly describes psychological distress and trauma resulting from wrongful arrests and detentions.

Directly caused: Victims like PERSON_001 and PERSON_003 experienced severe distress and trauma, described as 'a nightmare', from being wrongfully arrested and imprisoned for up to six nights.

Inferred additional harm: It is likely that many of the other 140 wrongfully stopped or arrested individuals experienced significant anxiety, distress, and psychological trauma due to police intervention.

People affected

• Occurrences reported: 1
• People reportedly harmed: 140
• People reportedly exposed: 1840

Potential causes

Management

• Lack of Operational Transparency: Management failed to provide details on data storage and algorithms.
• Inadequate Access Controls: Ministry of Security allowed unauthorized queries on biometric data.

Technology

• Facial Recognition False Positives: The software incorrectly matched innocent citizens to wanted criminals.
• System Vulnerability to Tampering: The system lacked safeguards to prevent unauthorized searches and surveillance.

Data Inputs

• Inaccurate Database Records: Errors in the underlying police database led to incorrect matches.
• Unauthorized Biometric Requests: Biometric data of journalists and public figures was improperly queried.

Human Factors

• Overreliance on System Outputs: Police arrested individuals based on matches without proper verification.

Process and Methods

• Lack of Independent Control: No independent mechanisms existed to audit system queries and matches.
• Opaque Software Procurement: Software standards and operational rules were not publicly disclosed.

Regulatory Environment

• Absence of Functioning Controls: Good legislation existed but lacked active, independent enforcement.

Information quality

• Classification confidence: High
• Reason for confidence: The report is highly detailed, based on a journalistic investigation and official judicial reports. It provides specific numbers of victims (140), names of key figures, and details of the legal proceedings. The role of the AI system (facial recognition) is explicitly stated, and the harms (wrongful arrests, privacy violations) are directly documented.
• Ambiguities identified: The specific algorithm and developer of the software remain undisclosed, and the exact technical cause of the database errors is not fully explained.
• Alternative interpretations: None. The facts of the wrongful arrests and the judicial suspension of the system are clear and uncontested.

Deployment of a facial recognition system in Buenos Aires resulted in 140 wrongful arrests due to false positives and unauthorized surveillance of journalists. The incident highlights significant human rights and societal stability concerns regarding unregulated biometric surveillance, leading to judicial intervention and the system's deactivation.

• Overall national security impact: Substantial
• Scope: Single nation
• Primary target: Argentina
• Alleged perpetrator: Ministry of Security of Buenos Aires

Threat characteristics

• Imminence: Long-term. The system remains deactivated under judicial order, representing an ongoing legal and regulatory debate rather than an active crisis.
• Autonomy: Human-supervised. The AI autonomously scanned video feeds and flagged matches, but human police officers executed the stops and arrests.
• Novelty: Established threat. Facial recognition false positives and misuse for surveillance are well-documented issues globally with established precedents.

Impact by dimension

• Physical security: Negligible. No physical infrastructure damage or kinetic threats occurred; the system was used for public surveillance and police checks.
• Information security: Minor. The Ministry of Security accessed biometric data of journalists investigating the system without authorization, representing a minor intelligence and privacy compromise.
• Sovereignty: Minor. Judicial intervention suspended the state-deployed system due to irregularities, representing minor disruption to localized government law enforcement functions.
• Economic security: Negligible. No significant threats to national economic stability or strategic technological competitive advantage were reported.
• Societal stability: Substantial. Faulty facial recognition led to 140 wrongful arrests and unauthorized surveillance of journalists, violating civil liberties and prompting judicial suspension of the system.