Facial Recognition Trial Performed Poorly at Notting Hill Carnival

The facial recognition trial by London’s Metropolitan Police Service at the Notting Hill Carnival reportedly performed poorly with a high rate of false positives.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The primary failure was the facial recognition system's extremely high false positive rate (98%) and general lack of accuracy under real-world crowd conditions.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Police deployed and defended an immature technology, relying on its outputs to stop and question innocent citizens.
• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: The system scanned the biometric data of up to 2 million attendees without consent and matched them against unlawfully retained custody databases.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the deployment of an automated facial recognition system that generated unexpected false matches, leading to wrongful police stops.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk (Level 2). The reports describe the deployment of 'Facial Recognition Technology' by the police for 'law enforcement applications, including facial recognition' to identify suspects in public crowds, which is explicitly classified as a high-risk application under the EU AI Act due to its significant implications for fundamental rights and public safety.

AI system and alleged parties

• AI system: automated facial recognition (AFR) technology
• AI purpose: Face Recognition; Identification
• Behaviour type: Assistant
• Alleged developer: unknown
• Alleged deployer: Metropolitan Police Service
• Alleged harmed parties: Notting Hill Carnival goers

Harm severity

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Minor, indirect Severe
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Substantial, indirect Severe
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: Yes, the reports describe concerns about demographic accuracy and potential racial bias in the facial recognition software.

Directly caused: The commercial software used by the police had not been tested for demographic accuracy biases, and police resisted recording ethnicity figures for identified individuals.

Indirectly caused: N/A

Inferred additional harm: Given known demographic biases in facial recognition, minority groups at the carnival (a celebration of black British culture) likely faced disproportionate rates of false matches and police stops.

Civil rights

Reported: Yes, the reports explicitly describe human rights concerns raised by campaign groups regarding public surveillance.

Directly caused: Campaigners stated that real-time scanning of faces in a crowd without knowledge breaches human rights laws and may dissuade people from attending public events.

Indirectly caused: The retention of over 20 million custody images, including of innocent people, was ruled unlawful by the High Court in 2012, violating civil liberties.

Inferred additional harm: The deployment of inaccurate surveillance technology in public spaces likely infringed upon the freedom of assembly and expression for the thousands of attendees.

Privacy

Reported: Yes, the reports explicitly describe privacy violations and concerns.

Directly caused: The reports state that the use of facial images in public places is highly intrusive of individual freedom because images are captured without the subjects' awareness or consent.

Indirectly caused: The unlawful retention of millions of facial images on police databases violates the privacy expectations of innocent members of the public.

Inferred additional harm: Up to 2 million carnival attendees had their biometric data captured and processed without their explicit consent, representing a widespread, unconsented intrusion of privacy.

People affected

• Occurrences reported: 1
• People reportedly harmed: 35
• People reportedly exposed: 2000000

Potential causes

Management

• Premature Technology Deployment: Police leadership deploying unproven systems in large public gatherings.
• Defensive Posture on Failures: Management framing high false-positive rates as operational successes.

Technology

• Immature Matching Algorithms: Poor match rates in crowded public spaces compared to controlled environments.
• Lack of Demographic Bias Testing: Commercial software not tested for racial or demographic accuracy biases.

Data Inputs

• Outdated Watch List Data: Suspect databases not updated, leading to stops of already-processed people.
• Unlawful Custody Image Retention: Over 20 million custody images stored, including those of innocent people.
• Lack of Standardized Database System: No single, shared system for storing and searching police-held images.

Human Factors

• Automation Bias and Overtrust: Humans tend to trust algorithmic outputs without considering the consequences.
• Intrusive Public Stops: Officers stopping and questioning citizens based solely on system alerts.

Process and Methods

• Inadequate Trial Evaluation: Lack of proper evaluation, peer review, and publication of trial results.
• No Mental Health Consultation: Watch lists compiled without consulting mental health professionals.

Regulatory Environment

• Delayed National Biometrics Strategy: Government biometrics strategy delayed, leaving a policy vacuum.
• Lack of Legislative Framework: No Parliament-approved laws or independent oversight for biometrics.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the 2017 Notting Hill Carnival trial, including specific statistics on false positive rates (98%), the number of false matches (35), and the lack of legal/regulatory frameworks. Multiple sources (government, news, tech media) corroborate the core facts of the incident, leaving little ambiguity about the system's performance and the resulting civil rights concerns.
• Ambiguities identified: There is minor disagreement on whether the stopped individuals were technically 'arrested' or merely 'stopped and questioned', but this does not affect the overall classification.

The 2017 Metropolitan Police facial recognition trial at the Notting Hill Carnival demonstrated high failure rates, raising localized civil liberties and privacy concerns. However, the incident posed negligible threats to broader national security, critical infrastructure, or state sovereignty.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Single nation
• Primary target: United Kingdom
• Alleged perpetrator: Metropolitan Police Service

Threat characteristics

• Imminence: Long-term. The incident represents a historical trial and ongoing policy debate rather than an active, immediate crisis.
• Autonomy: Human-supervised. The AI scanned and flagged matches, but human police officers retained control over final verification and intervention decisions.
• Novelty: Evolved capability. Real-time biometric scanning of large public crowds represented an evolution of existing static database matching techniques.

Impact by dimension

• Physical security: Negligible. No physical threats, kinetic actions, or critical infrastructure disruptions occurred during this public surveillance trial.
• Information security: Negligible. The incident did not involve intelligence compromises, classified data leaks, or foreign information warfare operations.
• Sovereignty: Negligible. The trial was a domestic law enforcement operation and did not threaten state sovereignty, border control, or electoral systems.
• Economic security: Negligible. There was no theft of strategic technology, impact on critical supply chains, or disruption of financial systems.
• Societal stability: Minor. The deployment of inaccurate facial recognition scanned thousands without consent, raising localized civil liberties and privacy concerns.