NYC Subway AI Weapons Scanners Yield High False Positive Rate and Detect No Guns in Month-Long Pilot Test

NYC implemented an AI enabled weapons scanner for a month-long pilot with limited success. Despite not finding any weapons during the September 2024 testing phase, there were 118 false positives in which a person was searched under suspicion of carrying a weapon with no actual gun detections.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The Evolv scanner failed to perform reliably under subway conditions, resulting in a high false positive rate and zero weapon detections, demonstrating a clear lack of capability and robustness.

Additional risk subdomains

• 4.3 Fraud, scams, and targeted manipulation: Evolv engaged in misleading marketing practices, overstating the AI's capabilities to sell the scanners at up to five times the rate of standard metal detectors.
• 5.1 Overreliance and unsafe use: City officials deployed the technology in a critical public transit environment despite explicit warnings from the developer's CEO that subways were an unsuitable use case.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was primarily caused by the AI system's failure to perform reliably in a deployed environment, generating high rates of false positives and failing to detect actual weapons.

EU AI Act risk tier

• Risk tier: 2 High Risk

Risk Level 2 (High Risk): The report describes an AI system used in law enforcement applications and critical public infrastructure (subway transit), which are explicitly classified as High Risk under the EU AI Act.

AI system and alleged parties

• AI system: Evolv Express (Evolv)
• AI purpose: Threat Detection; Smart Surveillance
• Behaviour type: Autonomous
• Alleged developer: Evolv Technology
• Alleged deployer: New York City Government
• Alleged harmed parties: New York City subway riders

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Substantial, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: Yes, the 118 passengers who triggered false positives were singled out for police searches.

Directly caused: 118 passengers were subjected to differential treatment in the form of police searches and transit delays due to false positive alerts from the AI scanner.

Civil rights

Reported: Yes, civil liberties groups argued that scanning millions of riders was unconstitutional.

Directly caused: 118 passengers had their civil liberties and privacy temporarily infringed upon by being subjected to unwarranted police searches based on false positive alerts.

Privacy

Reported: Yes, passengers were scanned in public transit stations.

Directly caused: 2,749 passengers were subjected to AI-powered scans without explicit individual consent during the subway pilot program.

Epistemic

Reported: Yes, Evolv engaged in misleading marketing claims about their technology's capabilities.

Directly caused: Evolv repeatedly made misleading claims in marketing materials regarding their system's ability to detect weapons and ignore harmless items, deceiving school districts and public officials.

People affected

• Occurrences reported: 1
• People reportedly harmed: 118
• People reportedly exposed: 2749

Potential causes

Management

• Prioritizing Hype Over Safety: Company and city officials prioritized high-tech marketing over accuracy.
• Ignoring Internal Warnings: Management deployed tech despite the CEO stating subways are a bad use case.
• Misleading Investor Relations: Executives overstated capabilities, leading to shareholder lawsuits.

Technology

• Subway Electromagnetic Interference: Railway interference degraded the accuracy of the scanner sensors.
• Basic Metal Detection Limitations: The underlying tech functions as a metal detector, not a weapon sensor.
• Flawed AI Classification: AI algorithms failed to distinguish weapons from harmless metal objects.

Data Inputs

• Lack of Subway-Specific Data: System was not trained on data reflecting subway environmental noise.
• Inadequate Signal Filtering: Sensor inputs failed to filter out common metallic items in transit.
• Unrefined Sensor Input Data: Sensor inputs were triggered by harmless personal metallic items.

Human Factors

• Political Overconfidence: The NYC Mayor pushed deployment despite technical warnings from the CEO.
• Operator Alert Fatigue: Over 100 false alarms created unnecessary searches and officer burden.
• User Misreliance on Marketing: Buyers relied on overstated marketing claims rather than empirical data.

Process and Methods

• Inadequate Pilot Evaluation: The 30-day pilot lacked rigorous pre-deployment testing and metrics.
• Deceptive Marketing Practices: Sales teams overstated weapon detection accuracy and capabilities.
• Lack of Independent Validation: No third-party testing was done to verify claims before subway trial.

Regulatory Environment

• Lax Pre-Market AI Oversight: No regulatory standards required proof of AI efficacy before public use.
• FTC and SEC Interventions: Regulatory bodies acted post-deployment rather than preventing false claims.
• Lack of Public Transit Standards: No specific safety or accuracy mandates exist for transit AI scanners.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide official statistics from the NYPD (2,749 scans, 118 false positives) and clear details regarding the FTC settlement with Evolv. The facts are consistent across both sources, leaving little ambiguity about the system's performance and the regulatory actions taken.
• Ambiguities identified: The exact nature of the '12 knives' detected (whether they were illegal or allowed pocket knives) is not specified by the police.

A pilot program of Evolv's AI weapons scanners in the NYC subway system failed to detect any firearms while generating 118 false positives. The incident highlights minor national security concerns regarding the reliability of AI in critical infrastructure security and the risks of misleading marketing influencing public procurement.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Evolv

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding AI efficacy and procurement rather than an active crisis requiring immediate response.
• Autonomy: Human-supervised. The AI system autonomously flagged targets, but human police officers made the final decision to conduct searches.
• Novelty: Established threat. False positives and performance failures in automated security screening technologies are well-established issues.

Impact by dimension

• Physical security: Minor. The AI scanner failed to detect weapons in a critical public transit environment, creating potential security vulnerabilities, though no kinetic attacks occurred during the pilot.
• Information security: Negligible. No intelligence compromise, classified data theft, or systematic information warfare operations were associated with this deployment.
• Sovereignty: Minor. Local government decision-making was influenced by misleading marketing to deploy ineffective technology in public transit, but core sovereignty was not threatened.
• Economic security: Minor. Public entities overpaid for ineffective technology due to misleading AI marketing, representing financial waste but not a systemic threat to economic security.
• Societal stability: Minor. 118 passengers experienced false positives leading to unwarranted searches, representing minor, localized civil liberties concerns but no mass societal unrest.