Live facial recognition is tracking kids suspected of being criminals

Buenos Aires city government uses a facial recognition system that has led to numerous false arrests.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The facial recognition system performs significantly worse on children than adults, performing worse by a factor of six on kids, which subjects them to a disproportionate risk of false arrest.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The system suffers from a general lack of robustness, leading to numerous false matches and arrests of adults as well as children.
• 5.1 Overreliance and unsafe use: Police officers rely on the system's real-time alerts to make arrests without established verification protocols or error-correction mechanisms.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The harm was triggered by the live facial recognition system's algorithmic false matches, which was an unexpected outcome of deploying the system for law enforcement.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes a law enforcement application of facial recognition, which is explicitly classified as a High Risk system under the EU AI Act definitions.

AI system and alleged parties

• AI system: live facial recognition system
• AI purpose: Face Recognition; Identification
• Behaviour type: Autonomous
• Alleged developer: unknown
• Alleged deployer: Buenos Aires city government
• Alleged harmed parties: Buenos Aires children

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Substantial, indirect Substantial
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Severe
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment of children due to the system's performance disparities.

Directly caused: Children are subjected to a significantly higher rate of misidentification and false arrest because the algorithm performs six times worse on kids than adults.

Inferred additional harm: Other demographic groups, such as women or ethnic minorities, may also face higher rates of misidentification, as is common with facial recognition systems, though not detailed in this report.

Civil rights

Reported: The report explicitly describes violations of human and civil rights, including the right to privacy, freedom of expression, and protection of children's rights.

Directly caused: Innocent citizens were falsely arrested and detained, violating their right to liberty, and children's private details were illegally exposed in a public database.

Indirectly caused: The pervasive surveillance and risk of false arrest create a chilling effect that restricts citizens' freedom of expression and movement in public spaces.

Inferred additional harm: Numerous undocumented false stops and detentions by police officers acting on erroneous real-time alerts from the system.

Privacy

Reported: The report explicitly describes severe privacy violations affecting both children and the general public.

Directly caused: The personal data of at least 166 children, including full names and national IDs, was exposed in an unprotected plain-text spreadsheet online.

Indirectly caused: Millions of subway commuters, including children, are subjected to invasive real-time biometric scanning without their consent or knowledge.

Inferred additional harm: The systematic collection, matching, and storage of biometric data of the city's population without robust security or oversight frameworks.

Psychological

Reported: The report explicitly describes psychological harm, including distress from false detentions and a chilling effect on children's behavior.

Directly caused: Individuals suffered severe distress and anxiety from being falsely arrested and detained, such as one man who was held for six days.

Indirectly caused: Children experience a chilling effect on their freedom of expression, leading them to become extremely self-censoring and hyper-vigilant about their public behavior.

Inferred additional harm: Widespread anxiety and loss of trust among the public, particularly families with children who are aware of the system's high error rates and lack of correction mechanisms.

People affected

• Occurrences reported: 1
• People reportedly harmed: 168
• People reportedly exposed: 10000

Potential causes

Management

• Inadequate Procurement Testing: Tested only on adult city employees before being deployed.
• Lack of Oversight: Management failed to monitor database accuracy and system integration.

Technology

• Adult-Only Algorithm Training: Algorithm was trained and tested primarily on adult faces, not children.
• High Error Rate on Minors: Performance is six times worse on children than on adults.

Data Inputs

• Flawed CONARC Database: Contains typos, conflicting details, and outdated photo IDs.
• Illegal Inclusion of Minors: Children's details and names are illegally included in the database.
• Unsecured Data Spreadsheet: Plain-text spreadsheet without password protection is publicly searchable.

Human Factors

• Operator Overreliance on System: Police arrest flagged individuals without verifying identity.
• Lack of Technical Understanding: Government procured technology without understanding human rights implications.

Process and Methods

• No Error Correction Mechanism: No way to correct mistakes in the algorithm or database once flagged.
• No Police Handling Protocols: Police have no established protocols for handling false arrests.
• Lack of Public Consultation: System was implemented without public feedback or trial transparency.

Regulatory Environment

• Lack of Legal Framework: System deployed before legislation to legalize facial recognition was passed.
• Privacy Law Violations: Inclusion of minor names violates international human rights privacy laws.

Information quality

• Classification confidence: High
• Reason for confidence: The report is highly detailed, drawing on a formal investigation by Human Rights Watch. It provides specific dates, database names, and concrete case studies of individuals affected, allowing for highly confident classifications.
• Ambiguities identified: The exact total number of false arrests is not quantified, described only as 'numerous false arrests'.
• Alternative interpretations: None, the facts clearly point to a systemic failure of a deployed facial recognition system.

The deployment of a flawed live facial recognition system in Buenos Aires, linked to an exposed national criminal database, resulted in systemic human rights violations and false arrests. While not a kinetic or international threat, the incident represents a substantial domestic security concern regarding data protection, algorithmic bias against minors, and the erosion of civil liberties.

• Overall national security impact: Substantial
• Scope: Single nation
• Primary target: Argentina
• Alleged perpetrator: Buenos Aires city government

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing, systemic domestic policy and technology governance issue rather than an immediate national security crisis.
• Autonomy: Human-supervised. The AI system autonomously scans feeds and flags matches, but human police officers make the final decision to execute detentions.
• Novelty: Established threat. Biometric surveillance errors, algorithmic bias against minors, and database security failures are well-documented and established risks.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic attacks, or critical infrastructure disruptions were reported in connection with this incident.
• Information security: Minor. The national criminal database (CONARC) was severely compromised, existing as an unprotected, plain-text spreadsheet accessible via Google Search, representing a significant data security failure.
• Sovereignty: Minor. Local government and law enforcement functions were disrupted by flawed automated decision-making, leading to erroneous detentions without proper verification protocols.
• Economic security: Negligible. No significant threats to economic stability, strategic industries, or technological competitive advantage were identified.
• Societal stability: Substantial. The deployment of live facial recognition led to systematic privacy violations, illegal exposure of minors' data, and multiple false arrests, causing a chilling effect on public freedoms.