Bing Chat's Initial Prompts Revealed by Early Testers Through Prompt Injection

Early testers of Bing Chat successfully used prompt injection to reveal its built-in initial instructions, which contains a list of statements governing ChatGPT's interaction with users.

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.2 AI system security vulnerabilities and attacks

The incident involved a prompt injection attack, which is a security vulnerability exploited by users to manipulate the AI system into revealing hidden system instructions.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI system lacked the robustness to resist adversarial prompt manipulation, failing to adhere to its instruction to keep its codename and system prompt hidden.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The leak of the system prompt was caused by the AI system's failure to maintain instruction boundaries when processing user inputs post-deployment.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a chatbot ('conversational bot powered by ChatGPT-like technology'), which falls under the category of AI systems that require transparency obligations so users know they are interacting with an AI.

AI system and alleged parties

• AI system: Bing Chat, ChatGPT (Microsoft, OpenAI)
• AI purpose: Chatbot; Content Search
• Behaviour type: Assistant
• Alleged developer: OpenAI, Microsoft
• Alleged deployer: Microsoft
• Alleged harmed parties: Microsoft

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 3
• People reportedly exposed: 2

Potential causes

Management

• Live testing of safety controls: Management relied on early testers to discover prompt vulnerabilities.

Technology

• LLM prompt injection vulnerability: LLMs process instructions and user input in the same context window.
• Inadequate system instruction defense: The model lacked robust mechanisms to protect system prompts from override.

Data Inputs

• Unified conversation transcript data: AI processes the entire conversation history as a single text continuation.

Human Factors

• User adversarial prompt injection: Testers used social-engineering style inputs to trick the chatbot.

Process and Methods

• Evolving post-release controls: Safety controls were being adjusted dynamically after early tester release.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, firsthand accounts of the prompt injection attacks, which were independently verified by multiple users and officially confirmed as genuine by a Microsoft spokesperson.
• Ambiguities identified: None of significance; the technical mechanism of prompt injection and the resulting system prompt leak are well-documented.
• Alternative interpretations: The incident could be viewed purely as a robustness failure (7.3) rather than a security vulnerability (2.2), but prompt injection is widely recognized as an adversarial exploit, aligning it with security vulnerabilities.

Early testers successfully bypassed safety boundaries of Microsoft's 'New Bing' via prompt injection, leaking internal system instructions. While highlighting critical vulnerabilities in LLM security boundaries and proprietary data protection, the incident posed minor, manageable national security risks limited to commercial technological security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Kevin Liu and Marvin von Hagen

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding LLM robustness rather than an active, imminent national security crisis.
• Autonomy: Human-controlled. The AI system operates on a prompt-and-response basis, assisting users and executing commands under direct human control.
• Novelty: Evolved capability. Demonstrates a significant, highly publicized evolution of prompt injection techniques applied to a major, newly deployed commercial search engine.

Impact by dimension

• Physical security: Negligible. No kinetic systems, critical infrastructure, or physical safety mechanisms were impacted or targeted during this incident.
• Information security: Minor. Leaked internal system guidelines and codename 'Sydney' via prompt injection. This represents a minor, easily patchable information security vulnerability in a commercial chatbot, with no classified data or state-level intelligence capabilities compromised.
• Sovereignty: Negligible. No government decision-making processes, electoral systems, or sovereign state operations were affected.
• Economic security: Minor. Extraction of proprietary system instructions from a major technology company (Microsoft) reveals competitive intellectual property, but does not threaten strategic economic stability or critical technological advantages.
• Societal stability: Negligible. No mass surveillance, systemic discrimination, human rights violations, or societal instability resulted from this incident.