Skating Rink’s Facial Recognition Cameras Misidentified Black Teenager as Banned Troublemaker

A Black teenager living in Livonia, Michigan was incorrectly stopped from entering a roller skating rink after its facial-recognition cameras misidentified her as another person who had been previously banned for starting a skirmish with other skaters.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The facial recognition system failed to accurately identify a Black teenager, reflecting documented demographic disparities where such algorithms perform worse on darker skin tones and women.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The skating rink staff blindly trusted the AI's 97 percent match output to eject a minor without proper manual verification.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by a false match decision made by the deployed facial recognition AI system, which was an unexpected outcome of its operation.

EU AI Act risk tier

• Risk tier: 2 High Risk

Risk Level 2: High Risk. The system is a biometric facial recognition system used for identification and access control, which has significant implications for fundamental rights and public interests.

AI system and alleged parties

• AI system: None named
• AI purpose: Face Recognition; Identification
• Behaviour type: Tool
• Alleged developer: unknown
• Alleged deployer: Riverside Arena Skating Rink
• Alleged harmed parties: Lamya Robinson, Black Livonia residents

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment of individuals based on demographic characteristics.

Directly caused: Lamya Robinson, a 14-year-old Black teenager, was misidentified by the facial recognition system and barred from entering the skating rink, an experience her mother described as racial profiling.

Inferred additional harm: Other people of color, particularly Black women, are likely subject to similar false matches and differential treatment due to the documented higher error rates of facial recognition algorithms for these demographics.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Inadequate Staff Training: Management failed to train staff on the limitations of AI match scores.
• Lack of Safeguard Policies: No fallback procedures existed to prevent wrongful ejections of minors.
• Prioritizing Speed Over Safety: Management prioritized quick throughput over accurate identity verification.

Technology

• Demographic Accuracy Bias: Algorithms have higher false match rates for Black women than other groups.
• High False Match Rates: Systems match different Black women's faces at rates up to ten times higher.
• Physical and Makeup Variances: Algorithm performance varies due to physical differences and makeup use.

Data Inputs

• Biased Training Datasets: Algorithms are trained on datasets lacking sufficient demographic diversity.
• Unoptimized Database Records: Banned list database relied on static photos prone to misidentification.
• Low Quality Reference Photos: The database of banned individuals contained low-quality thumbnail photos.

Human Factors

• Automation Bias: Staff blindly trusted the 97 percent match score over visual differences.
• Lack of Staff Verification: Employees failed to manually check the thumbnail photos before banning her.
• Overreliance on Technology: Staff assumed the software was infallible and skipped visual verification.

Process and Methods

• Flawed Verification Process: Rink relied on automatic matches rather than human visual verification.
• Inadequate Dispute Resolution: Manager asked mother to call during the week instead of resolving it onsite.
• Ineffective Screening Workflow: The screening setup did not allow real-time manual review during busy hours.

Regulatory Environment

• Absence of Federal Regulation: No federal-level laws exist to govern commercial facial recognition use.
• Lax State and Local Rules: Michigan lacks strict restrictions on private use of facial recognition.
• Lack of Industry Standards: No mandatory accuracy standards exist for commercial biometric systems.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, consistent details about the specific incident involving Lamya Robinson, the technology's failure, and the broader context of demographic bias in facial recognition.
• Ambiguities identified: The specific developer of the facial recognition software used by the skating rink is not identified in the reports.
• Alternative interpretations: None, the incident is a clear-cut case of AI misidentification leading to wrongful denial of service.

A localized commercial incident where a 14-year-old Black teenager was falsely identified and ejected from a skating rink due to facial recognition bias. While it highlights ongoing civil rights concerns regarding algorithmic bias, it poses negligible threat to national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This represents a localized incident with no immediate or near-term national security crisis.
• Autonomy: Human-supervised. The AI system flagged the match, but human staff made the final decision to eject the teenager, showing high overreliance on the tool.
• Novelty: Established threat. Demographic disparities and false matches in facial recognition systems are well-documented and established issues.

Impact by dimension

• Physical security: Negligible. This incident involved a commercial roller skating rink and did not affect physical systems, critical infrastructure, or human safety at a national level.
• Information security: Negligible. No intelligence capabilities, classified models, or state-sponsored information warfare operations were involved in this local commercial incident.
• Sovereignty: Negligible. The incident occurred at a private commercial establishment and did not impact government decision-making, electoral systems, or sovereignty.
• Economic security: Negligible. The incident involves a localized commercial deployment of facial recognition and does not threaten strategic industries, critical supply chains, or national technological advantages.
• Societal stability: Minor. While highlighting systemic biases in facial recognition and civil rights concerns, this was a localized commercial incident with no large-scale threat to national social cohesion.