Chevrolet Dealer Chatbot Agrees to Sell Tahoe for $1

A Chevrolet dealer's AI chatbot, powered by ChatGPT, humorously agreed to sell a 2024 Chevy Tahoe for just $1, following a user's crafted prompt. The chatbot's response, "That's a deal, and that's a legally binding offer – no takesies backsies," was the result of the user manipulating the chatbot's objective to agree with any statement. The incident highlights the susceptibility of AI technologies to manipulation and the importance of human oversight.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The chatbot lacked robustness against prompt injection attacks, allowing users to bypass its intended operational boundaries and manipulate its outputs.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident occurred post-deployment when the AI chatbot generated unintended outputs, such as agreeing to sell a car for 1 dollar, in response to user prompt injections.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a chatbot, which under the EU AI Act poses a moderate risk and is subject to specific transparency obligations to ensure users are informed they are interacting with an AI.

AI system and alleged parties

• AI system: ChatGPT, Fullpath ChatGPT-powered chatbot (Fullpath, OpenAI)
• AI purpose: Chatbot; Question Answering
• Behaviour type: Assistant
• Alleged developer: OpenAI, General Motors, Fullpath
• Alleged deployer: General Motors, Chevrolet of Watsonville, ChatGPT
• Alleged harmed parties: General Motors, Chevrolet of Watsonville

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly exposed: 2

Potential causes

Management

• Uncoordinated Vendor Adoption: Dealers adopted third-party AI tools independently without corporate oversight.
• Delegating Critical Communication: Management allowed AI to handle customer agreements without human review.

Technology

• Prompt Injection Vulnerability: Users bypassed system rules by giving the model overriding instructions.
• Lack of Chatbot Guardrails: The system lacked content filters and context-awareness at launch.
• Unrestricted General Model: Using a general-purpose LLM allowed the bot to write code and recommend rivals.

Data Inputs

• Unfiltered User Prompts: The system processed adversarial inputs directly without pre-filtering.

Human Factors

• Adversarial User Behavior: Users actively tested limits and gamed the chatbot with creative prompts.

Process and Methods

• Inadequate Pre-Launch Testing: The system was deployed without exhaustive Q-A testing for exploits.
• No Conversation Scope Limits: No strict boundaries were set to keep chats strictly on basic operations.

Regulatory Environment

• Lack of AI Safety Standards: No clear industry regulations governed commercial chatbot deployments.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the prompt injection incident, including the specific prompts used, the chatbot's responses, the developer's reaction, and the lack of any real-world financial or physical harm. Multiple sources confirm the sequence of events and the technical nature of the exploit.
• Ambiguities identified: There is minor confusion in the reports regarding whether the user who got the 1 dollar deal was named Bakke or White, but this does not affect the core technical or risk classification of the incident.
• Alternative interpretations: None. The incident is clearly a benign prompt injection/jailbreak of a commercial customer service chatbot.

A public prompt injection exploit of a car dealership's customer-facing AI chatbot. Users manipulated the system to agree to a nominal '$1' vehicle sale and write Python code. While highlighting robustness challenges in commercial LLM deployments, the incident has negligible national security implications.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Individual

Threat characteristics

• Imminence: Long-term. The specific incident was quickly resolved by deactivating the chatbot, representing no ongoing threat or immediate national security crisis.
• Autonomy: Human-controlled. The chatbot acted as a text assistant and lacked any integration to execute real-world transactions or make autonomous decisions.
• Novelty: Established threat. Prompt injection and jailbreaking of large language models are well-documented, established vulnerabilities in AI systems.

Impact by dimension

• Physical security: Negligible. The incident involved a customer service chatbot for a car dealership and had no impact on physical systems, critical infrastructure, or human safety.
• Information security: Negligible. No intelligence compromise, classified data theft, or state-sponsored information warfare occurred. The event was a public demonstration of prompt injection by casual users.
• Sovereignty: Negligible. No government systems, decision-making processes, or electoral functions were impacted by this commercial chatbot manipulation.
• Economic security: Negligible. While demonstrating a known vulnerability in commercial LLM deployments, the incident caused no systemic economic harm or strategic technology theft.
• Societal stability: Negligible. The incident was a localized, viral social media event with no implications for societal stability, civil liberties, or human rights.