Tesla Autopilot Misidentified On-Road Horse-Drawn Carriage

Tesla Autopilot's computer vision system was shown in a video mistaking a horse-drawn carriage for other forms of transport such as a truck, a car, and a human following a car.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The Tesla Autopilot system failed to perform robustly when encountering out-of-distribution data (a horse-drawn carriage), leading to erratic visualization and potential safety risks.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The perception failure was caused by the Tesla Autopilot AI system's inability to correctly classify a horse-drawn carriage, which was an unintentional error occurring post-deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used in critical transport infrastructure. The report describes Tesla's Autopilot and Full Self-Driving technologies, which are AI systems in critical transport infrastructure where failures can have significant implications for safety.

AI system and alleged parties

• AI system: Autopilot (Tesla)
• AI purpose: Autonomous Driving; Image Classification
• Behaviour type: Autonomous
• Alleged developer: Tesla
• Alleged deployer: Tesla
• Alleged harmed parties: Tesla drivers, horse-drawn carriages

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly exposed: 2

Potential causes

Management

• Misleading Product Claims: Management promoted FSD as requiring no action, creating false expectations.

Technology

• OOD Generalization Failure: Deep learning models struggle with out-of-distribution data like carriages.
• Unstable Object Classification: System rapidly alternated between detecting a car, human, and truck.
• Erratic Visualization Rendering: System showed a ghost truck spinning, indicating tracking instability.

Data Inputs

• Lack of Training Data for Carriages: The model likely saw very few horse-drawn carriages during its training.
• Missing Object Class Definitions: Designers may not have included horse-drawn carriages in the target classes.

Human Factors

• Overreliance on Automation: Drivers may assume Autopilot can handle all road scenarios safely.

Process and Methods

• Inadequate Edge-Case Testing: Testing processes failed to validate performance against rare road obstacles.

Regulatory Environment

• Weak Marketing Oversight: Regulators like DMV noted misleading claims about self-driving capabilities.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent accounts of the incident, including a viral video showing the exact behavior of the Tesla Autopilot system when encountering a horse-drawn carriage. The technical cause (out-of-distribution data) is clear and agreed upon by observers.
• Ambiguities identified: It is unclear whether the Autopilot system would have taken incorrect physical actions (like braking or swerving) or if the error was limited to the visual display, as the driver remained in control.
• Alternative interpretations: None. The event is a straightforward perception failure of an autonomous driving system.

A Tesla Model Y Autopilot system failed to correctly identify a horse-drawn carriage in Switzerland, highlighting technical challenges with out-of-distribution data. The incident was a non-harmful consumer product glitch with negligible national security implications.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents a minor, post-deployment technical limitation rather than an active security crisis.
• Autonomy: Human-supervised. The Autopilot system operates with autonomous perception, but requires active human supervision and driver control.
• Novelty: Established threat. Perception failures and edge-case classification errors in autonomous driving systems are well-documented and established issues.

Impact by dimension

• Physical security: Negligible. Minimal impact. The incident was a perception glitch in a consumer vehicle's driver-assist system with no physical collision, injury, or critical infrastructure compromise.
• Information security: Negligible. No information warfare, intelligence compromise, or systematic information manipulation occurred during this incident.
• Sovereignty: Negligible. No threat to state authority, territorial control, or core government operations was posed by this consumer vehicle glitch.
• Economic security: Negligible. No threat to strategic industries, financial systems, or national technological competitive advantage, despite regulatory scrutiny of the developer.
• Societal stability: Negligible. No implications for societal stability, human rights, mass surveillance, or civil liberties.