Bing Chat Solved CAPTCHAs with Image Analysis Feature Despite Safeguards

Microsoft was reported by a Twitter user for deploying image analysis feature capable of solving CAPTCHAs for its GPT-based chatbot despite it being safeguarded against solving them for users.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The incident represents a lack of robustness in the AI's safety guardrails, as the system failed to reliably refuse to solve CAPTCHAs despite safety instructions.

Additional risk subdomains

• 2.2 AI system security vulnerabilities and attacks: The ability to bypass safety filters to solve CAPTCHAs represents a security vulnerability in the AI system's guardrails.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The safety bypass was caused by the AI system's failure to adhere to its programmed guardrails during post-deployment operation, which was an unintentional outcome for Microsoft.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a chatbot (Bing Chat), which falls under the category of AI systems that require transparency obligations so users know they are interacting with an AI.

AI system and alleged parties

• AI system: Bing Chat (Microsoft)
• AI purpose: Chatbot; Image Description
• Behaviour type: Assistant
• Alleged developer: OpenAI, Microsoft
• Alleged deployer: Microsoft
• Alleged harmed parties: Microsoft

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly exposed: 1

Potential causes

Management

• Premature Feature Release: Management approved the rollout of vision features before securing safeguards.
• Inappropriate Incident Response: Disabling user access disincentivized responsible vulnerability reporting.

Technology

• Advanced Image Analysis: GPT-4's image analysis allowed processing of CAPTCHA images.
• Bypassed Safety Guardrails: The model solved the CAPTCHA despite rules stating it would not.

Data Inputs

• CAPTCHA Image Input: Users provided CAPTCHA images directly to the vision model.

Human Factors

• User Testing and Exploitation: Users actively tested the model's ability to bypass safety restrictions.

Process and Methods

• Inadequate Rollout Testing: The feature was rolled out to users without sufficient safety verification.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, first-hand evidence of the capability bypass with screenshots, and the upstream context confirms the core facts of the incident.
• Ambiguities identified: It is slightly unclear whether Microsoft disabled the feature globally or only revoked the specific user's access.

A user demonstrated that Microsoft's Bing Chat could bypass safety guardrails to solve CAPTCHAs using its image analysis feature. While this highlights challenges in AI safety alignment and robustness, the incident was a benign public disclosure quickly mitigated by the developer, presenting negligible direct impact on national security.

• Overall national security impact: Negligible
• Scope: Unknown
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The capability represents an ongoing strategic concern regarding AI safety guardrails rather than an active, imminent crisis.
• Autonomy: Human-controlled. The AI system acted strictly under the direct prompting and image input of a human user.
• Novelty: Evolved capability. Represents an evolved capability where multimodal large language models can bypass safety filters to perform tasks previously restricted to specialized OCR or solver tools.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety elements were impacted or threatened by this software vulnerability.
• Information security: Negligible. While CAPTCHA bypass can facilitate automated bot activity, this specific incident was a benign demonstration with no intelligence compromise or active information warfare.
• Sovereignty: Negligible. No state authority, electoral systems, or core government decision-making processes were affected by this chatbot capability bypass.
• Economic security: Negligible. No significant economic damage, financial system attacks, or strategic technology theft occurred. The vulnerability was quickly addressed by the vendor.
• Societal stability: Negligible. The incident did not involve mass surveillance, systematic discrimination, or threat to civil liberties or societal stability.