Cruise AV Ran Over Fire Hose in Active Fire Scene

A Cruise AV ran over a fire hose that was being used in an active firefighting area.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The autonomous vehicles failed to perform reliably under complex real-world conditions, specifically active emergency scenes, demonstrating a lack of capability and robustness in critical situations.

Additional risk subdomains

• 6.5 Governance failure: Inadequate regulatory oversight and lack of data transparency from state regulators like the CPUC and DMV hindered the management of AI safety risks.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incidents were caused by autonomous vehicles failing to navigate emergency scenes correctly after being deployed on public roads, which represents an unintentional failure of the AI system.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The AI systems are deployed in critical infrastructure (traffic management and transport) and their failures directly impact emergency response services and public safety.

AI system and alleged parties

• AI system: Cruise AV (Cruise)
• AI purpose: Autonomous Driving; Navigation Assistant
• Behaviour type: Autonomous
• Alleged developer: Cruise
• Alleged deployer: Cruise
• Alleged harmed parties: San Francisco firefighters, San Francisco Fire Department

Harm severity

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 42
• People reportedly exposed: 150

Potential causes

Management

• Premature Commercial Deployment: Companies deployed vehicles before perfecting emergency scene interaction.
• Inadequate Safety Transparency: Management relied on trade secrets to avoid sharing safety performance data.

Technology

• Failure to Detect Emergency Scenes: AVs failed to recognize active fire scenes and avoid emergency apparatus.
• Sensor Vulnerability to Obstacles: Downed wires got tangled in rooftop sensors, immobilizing the vehicle.
• Inadequate Navigation Logic: AVs blocked firehouse driveways and stopped behind responding trucks.

Data Inputs

• Poor Sensor Detection of Hoses: Sensors and AI failed to identify flat fire hoses on the ground.

Human Factors

• Delayed Remote Operator Action: Remote operators could not quickly move stuck vehicles out of harm's way.
• Forced Physical Intervention: Firefighters had to shatter a window to stop the autonomous vehicle.

Process and Methods

• Ineffective Stalled Vehicle Removal: Firefighters had to wait for human employees to arrive and move cars.

Regulatory Environment

• Shielding of Safety Information: DMV agreement allowed companies to hide crash safety data from the public.
• Premature Operation Expansion: Regulators planned to allow unlimited robotaxis despite known issues.
• Regulatory Conflict of Interest: A commissioner previously served as the top attorney for Cruise.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, detailed accounts of specific incidents with dates, locations, and official statements from the San Francisco Fire Department, police union, and AV companies. The nature of the AI failures and their impacts are well-documented, leaving little ambiguity about the core events.
• Ambiguities identified: The exact technical reasons why the AV sensors and algorithms failed to recognize emergency scenes remain unspecified.

Fully autonomous commercial vehicles in San Francisco repeatedly interfered with emergency responders, including blocking fire trucks and driving over active hoses. While causing no casualties, these incidents highlight emerging physical security and critical infrastructure challenges as autonomous systems integrate into public spaces without robust edge-case handling.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing technological challenge and regulatory concern regarding autonomous vehicle integration rather than an active national security crisis.
• Autonomy: Full autonomy. The autonomous vehicles operate and navigate public streets independently without in-vehicle human drivers or direct human intervention.
• Novelty: Evolved capability. Represents an evolved capability where fully autonomous systems deployed at scale in complex urban environments create novel failure modes with emergency services.

Impact by dimension

• Physical security: Minor. Autonomous vehicles repeatedly interfered with emergency response operations, blocking fire trucks and driving over active hoses, presenting localized risks to public safety.
• Information security: Negligible. No information warfare, intelligence compromise, or systematic information manipulation occurred during these incidents.
• Sovereignty: Minor. Local government functions (emergency services) faced operational disruptions, prompting calls for regulatory changes, but core state authority remained unaffected.
• Economic security: Minor. Minor property damage to AVs and emergency equipment occurred, alongside regulatory friction for US autonomous vehicle competitiveness, but no strategic threat.
• Societal stability: Negligible. While causing public concern and tension between city officials and AV companies, the incidents did not threaten social cohesion or civil liberties.