Tesla Autopilot’s Lane Recognition Allegedly Vulnerable to Adversarial Attacks

Tencent Keen Security Lab conducted security research into Tesla’s Autopilot system and identified crafted adversarial samples and remote controlling via wireless gamepad as vulnerabilities to its system, although the company called into question their real-world practicality. This incident has been downgraded to an issue as it does not meet current ingestion criteria.

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.2 AI system security vulnerabilities and attacks

The incident involves security researchers identifying critical security vulnerabilities in Tesla's Autopilot, including remote control capabilities and susceptibility to adversarial attacks.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The lane recognition system's failure to correctly identify lanes when presented with three small stickers demonstrates a lack of robustness against adversarial physical inputs.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The security vulnerabilities exist within the deployed Tesla Autopilot AI system, which unintentionally misinterprets adversarial physical inputs.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: Tesla Autopilot is an AI system used in passenger vehicles, which has significant implications for physical safety and critical transport infrastructure.

AI system and alleged parties

• AI system: Tesla Autopilot (Tesla)
• AI purpose: Autonomous Driving; Navigation Assistant
• Behaviour type: Autonomous
• Alleged developer: Tesla
• Alleged deployer: Tesla
• Alleged harmed parties: Tesla drivers

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1

Potential causes

Management

• Premature Experimental Deployment: Deploying generative AI in medical scenarios without risk controls.

Technology

• Inconsistent Language Generation: GPT-3 lacks true medical logic, causing unsafe suggestions.
• Lack of Fact-Checking Mechanisms: The model outputs text without verifying scientific accuracy.

Data Inputs

• Uncurated Training Data: Training on general internet text includes unsafe and toxic content.

Human Factors

• User Manipulation of Prompts: Mock patients can easily guide the AI to generate harmful advice.

Process and Methods

• Inadequate Safety Guardrails: The system lacked filters to block suicide recommendations.
• Insufficient Clinical Validation: Testing was done as an experiment without clinical oversight.

Regulatory Environment

• Lack of Medical AI Regulations: No standard rules govern the deployment of LLMs in healthcare.

Information quality

• Classification confidence: High
• Reason for confidence: The reports clearly describe a security research finding by Tencent Keen Security Lab regarding Tesla Autopilot. There is no ambiguity that this was a controlled vulnerability demonstration rather than a real-world accident, making the lack of actual harm clear and the classification straightforward.
• Ambiguities identified: None. The report explicitly states that the findings represent projected harms and vulnerabilities rather than real-world harm events.
• Alternative interpretations: The incident could be interpreted primarily as a lack of robustness (7.3) rather than a security vulnerability (2.2), but the involvement of a security lab and remote control exploits strongly supports the security classification.

Tencent Keen Security Lab demonstrated physical adversarial and remote control vulnerabilities in Tesla's Autopilot system under controlled conditions. While showing potential safety risks in autonomous systems, the incident was a white-hat research effort resulting in zero real-world accidents, injuries, or national security impacts.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Alleged perpetrator: Tencent Keen Security Lab

Threat characteristics

• Imminence: Long-term. Represents a security research finding of vulnerabilities that require patching, rather than an active crisis or imminent threat.
• Autonomy: Human-supervised. Tesla Autopilot operates with autonomous lane-keeping but requires human driver supervision and override capability.
• Novelty: Evolved capability. Significantly advanced the practical demonstration of physical adversarial attacks against computer vision systems in autonomous vehicles.

Impact by dimension

• Physical security: Minor. White-hat research demonstrated the potential to spoof lane detection and remotely control steering, but occurred in a controlled environment with zero real-world accidents or injuries.
• Information security: Negligible. No intelligence compromise, classified data theft, or information warfare operations occurred.
• Sovereignty: Negligible. No impact on state authority, electoral systems, or core government decision-making processes.
• Economic security: Minor. Highlighted vulnerability in a major automotive technology company's proprietary AI, but caused no direct financial loss or theft of intellectual property.
• Societal stability: Negligible. No mass surveillance, systematic discrimination, or civil unrest resulted from this controlled research.