Picture of Woman on Side of Bus Shamed for Jaywalking

Facial recognition system in China mistakes celebrity's face on moving billboard for jaywalker

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI system failed to perform robustly under real-world conditions because it lacked liveness detection, mistaking a 2D advertisement on a moving bus for a 3D human pedestrian.

Additional risk subdomains

• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: The system publicly exposed the victim's name and partial government ID number on a public billboard.
• 3.1 False or misleading information: The system generated and publicly displayed a false accusation that the victim had committed a traffic violation.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the Intellifusion DeepEye AI system making an erroneous identification post-deployment, which was an unintended failure of its classification capabilities.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is a real-time biometric identification and law enforcement application used by state authorities for public surveillance and traffic management.

AI system and alleged parties

• AI system: facial recognition system
• AI purpose: Face Recognition; Smart Surveillance
• Behaviour type: Autonomous
• Alleged developer: Ningbo traffic police
• Alleged deployer: Ningbo traffic police
• Alleged harmed parties: Dong Mingzhu

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Privacy

Reported: Yes, the reports state that the system displayed the victim's face, name, and partial government ID number on a public screen.

Directly caused: The system exposed the personally identifiable information, including a partial government ID number, of one individual on a public billboard without her consent.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Rapid Surveillance Scaling: Prioritized rapid deployment over technical accuracy and safety.
• Vendor Quality Oversight: Police accepted the AI system without rigorous edge-case testing.

Technology

• Lack of Liveness Detection: System failed to distinguish 2D bus ad from a real 3D human.
• No Speed or Motion Filtering: System did not filter out faces moving at vehicle speeds.
• Database Mismatch Error: System misidentified the surname as Ju during database lookup.

Data Inputs

• Bus Advertisement Image: A large, high-resolution portrait on a moving bus was captured.
• Unvalidated Reference Database: Database entries contained errors leading to incorrect name mapping.

Human Factors

• Lack of Human Verification: No human operator verified the AI's match before public display.
• Overreliance on AI Accuracy: Operators assumed the facial recognition system was infallible.

Process and Methods

• Automated Public Shaming: Process immediately displays suspected offenders on public screens.
• Inadequate System Testing: System deployed without testing against common urban visual noise.

Regulatory Environment

• Weak Privacy Protections: Inadequate laws allowed rapid deployment of invasive surveillance.
• Absence of AI Standards: No regulatory standards required liveness checks in public systems.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly consistent across multiple reputable news outlets, providing clear details of the technical failure, the parties involved, and the resolution.

A municipal facial recognition system in China autonomously and incorrectly identified a prominent businesswoman from a bus advertisement as a jaywalker, displaying her details on a public shaming billboard. The incident highlights minor localized human rights and privacy concerns regarding automated surveillance systems but has negligible broader national security impact.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: China
• Alleged perpetrator: Ningbo Traffic Police

Threat characteristics

• Imminence: Long-term. The incident represents a localized software error and ongoing surveillance capability concern rather than an active national security crisis.
• Autonomy: Full autonomy. The AI system autonomously captured, processed, identified, and publicly displayed the target's information on a billboard without human verification.
• Novelty: Evolved capability. Demonstrates an evolution in surveillance capabilities where biometric errors are directly and automatically translated into public shaming actions.

Impact by dimension

• Physical security: Negligible. No physical security threat, kinetic impact, or critical infrastructure disruption was caused by this traffic enforcement system error.
• Information security: Negligible. No classified intelligence compromise or systematic information warfare campaign occurred; this was a localized municipal surveillance error.
• Sovereignty: Minor. Represents a minor malfunction of local government law enforcement technology, but did not disrupt core state authority or national decision-making.
• Economic security: Negligible. No strategic technology theft, financial system disruption, or loss of technological competitive advantage resulted from the incident.
• Societal stability: Minor. The incident involved public shaming and privacy exposure of an individual due to a surveillance system error, representing a minor localized impact on civil liberties.