Cruise's Autonomous Vehicles Allegedly Engaging in Risky Behavior Near Pedestrians

Cruise's driverless vehicles are under federal investigation for possibly failing to exhibit due caution around crosswalks and pedestrians, with reports including one severe injury incident.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The incident stems from the Cruise autonomous vehicles failing to perform reliably and safely around pedestrians under varying real-world road conditions.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The safety risks and pedestrian collisions were caused by the autonomous driving system of the deployed Cruise vehicles failing to operate safely in real-world environments.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The autonomous vehicles operate in public transport and critical road infrastructure, posing significant implications for public safety and physical health.

AI system and alleged parties

• AI system: Cruise autonomous vehicle (General Motors)
• AI purpose: Autonomous Driving; Navigation Assistant
• Behaviour type: Autonomous
• Alleged developer: Cruise
• Alleged deployer: Cruise
• Alleged harmed parties: pedestrians, General public

Harm severity

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Minor, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Physical

Reported: The report explicitly describes physical harm to people, noting that NHTSA is aware of four incidents, including two that resulted in injuries.

Directly caused: A Cruise vehicle came to a stop on top of a pedestrian who had been launched into its path, requiring rescue crews to lift the car off her, resulting in severe injuries.

Indirectly caused: N/A

Inferred additional harm: N/A

People affected

• Occurrences reported: 4
• People reportedly harmed: 2
• People reportedly exposed: 4

Potential causes

Management

• Aggressive Commercial Targets: Management prioritized rapid expansion and revenue over safety safeguards.
• Premature Fleet Scaling: Scaling driverless services in complex cities before resolving technical bugs.

Technology

• Risky Pedestrian Detection: Autonomous vehicles failed to exercise appropriate caution around pedestrians.
• Inadequate Emergency Stop: The vehicle came to a stop directly on top of the injured pedestrian.

Data Inputs

• Unexpected Input Trajectory: Sensors did not properly anticipate a pedestrian launched by another vehicle.
• Poor Crosswalk Processing: AV sensors struggled to process pedestrians entering or present in crosswalks.

Human Factors

• Erratic Human Driver Action: A hit-and-run driver struck the pedestrian, throwing her into the AV path.
• First Responder Interference: AVs stalled or blocked emergency crews trying to assist at the scene.

Process and Methods

• Insufficient Scenario Testing: Testing processes failed to account for complex multi-vehicle crash scenarios.
• Inadequate Fleet Monitoring: Operational protocols failed to prevent risky behavior in dense urban areas.

Regulatory Environment

• Delayed Federal Investigations: Regulators often failed to complete safety investigations in a timely manner.
• Lagging Safety Oversight: State DMV initiated investigations only after multiple safety incidents occurred.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, factual details from reputable regulatory bodies (NHTSA and California DMV) regarding the safety investigations and specific incidents involving Cruise vehicles.
• Ambiguities identified: The exact technical failure that caused the vehicle to stop on top of the pedestrian rather than backing away or stopping sooner is not fully detailed.
• Alternative interpretations: None, the safety risks and regulatory actions are clearly documented.

Cruise autonomous vehicles are under federal and state investigation following pedestrian accidents in San Francisco. The incident represents an unintentional AI safety failure with minor localized physical safety and economic impacts. It does not pose a significant threat to US national security and is being managed through standard regulatory frameworks.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This is an ongoing safety and regulatory oversight issue rather than an immediate national security crisis requiring urgent response.
• Autonomy: Full autonomy. The Cruise autonomous vehicles operate and make driving decisions in real-world traffic without direct human intervention.
• Novelty: Established threat. Autonomous vehicle safety failures, collisions, and subsequent regulatory investigations are established issues with existing precedents.

Impact by dimension

• Physical security: Minor. The incident involves localized safety failures of autonomous vehicles causing pedestrian injuries. While posing a public safety risk, it does not threaten critical infrastructure or constitute a kinetic military threat, and is manageable via standard regulatory procedures.
• Information security: Negligible. There is no indication of information warfare, intelligence compromise, or systematic information manipulation in this incident.
• Sovereignty: Negligible. Core government functions and sovereignty are unaffected; standard domestic regulatory bodies (NHTSA, California DMV) are actively managing the oversight process.
• Economic security: Minor. The investigation and fleet reduction impact Cruise's commercial operations and expansion plans, representing minor localized economic and technological setbacks for a single company.
• Societal stability: Negligible. The incidents represent localized traffic safety issues and do not threaten societal stability, civil liberties, or population-scale human rights.