Firefighters Smashed Cruise AV's Front Window to Stop It from Running over Fire Hoses

Local firefighters were only able to stop a Cruise AV from driving over fire hoses that were in use in an active fire scene when they shattered its front window.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The autonomous vehicles failed to perform reliably under complex real-world conditions, specifically failing to recognize and appropriately navigate active emergency scenes.

Additional risk subdomains

• 6.5 Governance failure: The reports highlight regulatory oversight failures, where state commissions allowed companies to redact safety data and push for rapid expansion despite local safety objections.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incidents were caused by the autonomous driving systems of Cruise and Waymo vehicles post-deployment, resulting in unintended interference with emergency services.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The AI systems are deployed in critical infrastructure (road traffic management and transport) and their failure directly impacts public safety and emergency response services.

AI system and alleged parties

• AI system: Cruise AV
• AI purpose: Autonomous Driving; Navigation Assistant
• Behaviour type: Autonomous
• Alleged developer: Cruise
• Alleged deployer: Cruise
• Alleged harmed parties: San Francisco residents, San Francisco firefighters, San Francisco Fire Department

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 41
• People reportedly exposed: 100

Potential causes

Management

• Prioritizing Rapid Expansion: Management pushed for 24/7 unlimited fleet expansion despite safety risks.
• Inadequate Safety Transparency: Management submitted heavily redacted safety reports to regulators.

Technology

• Failure to Detect Fire Hoses: AV sensors and perception failed to recognize fire hoses on the ground.
• Inability to Detect Fire Scenes: AI planning failed to recognize active emergency zones and avoid them.
• Difficult Remote Disengagement: Only specialized Cruise experts could disengage the autonomous mode.

Data Inputs

• Lack of Real-Time Emergency Data: System lacked real-time feeds on active emergency scenes and road closures.
• Rooftop Sensor Obstruction: Downed electrical wires tangled in rooftop sensors, disrupting data input.

Human Factors

• No On-Board Driver for Manual Move: Lack of an in-vehicle driver forces reliance on slow remote assistance.
• First Responder Communication Gap: Firefighters had to smash a window to stop the vehicle due to no interface.

Process and Methods

• Inadequate Emergency Protocols: Lack of rapid physical override methods for first responders on scene.
• Bypassing Incremental Testing: Companies sought rapid expansion bypassing city incremental testing steps.

Regulatory Environment

• Broad Trade Secret Protections: DMV allowed companies to redact safety data, limiting public oversight.
• Regulatory and City Alignment Gap: State regulators approved expansion despite local first responder protests.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts from official city sources (SFFD, SFCTA) regarding multiple specific incidents of AV interference. The physical events, such as the window-smashing incident on January 21, 2023, are well-documented across multiple reports.
• Ambiguities identified: The exact technical reasons why the AV sensors and AI failed to detect active fire scenes or hoses are not fully explained.

Autonomous vehicles in San Francisco repeatedly interfered with emergency responders, including blocking fire trucks and driving near active fire hoses. While presenting localized public safety challenges and highlighting regulatory hurdles for AI integration, the incident has negligible direct national security implications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing regulatory and technical integration challenge for autonomous systems rather than an imminent national security crisis.
• Autonomy: Full autonomy. The vehicles were operating in fully driverless mode, making real-time navigation decisions without active human intervention.
• Novelty: Evolved capability. Represents an evolved challenge in autonomous system deployment as fully driverless fleets scale and interact with complex urban emergency scenarios.

Impact by dimension

• Physical security: Minor. Autonomous vehicles caused localized disruptions to emergency response services, presenting minor public safety risks but no systemic threat to critical infrastructure.
• Information security: Negligible. No information warfare, intelligence compromise, or systematic disinformation was associated with this incident.
• Sovereignty: Negligible. Local government operations faced minor municipal challenges, but there was no threat to state authority, territorial control, or core constitutional processes.
• Economic security: Negligible. The incident involved localized property damage and regulatory disputes, with no significant threat to national economic stability or strategic technology theft.
• Societal stability: Negligible. The incident did not involve mass surveillance, systematic discrimination, or population-scale social manipulation affecting societal stability.