Bing Chat Response Cited ChatGPT Disinformation Example

Reporters from TechCrunch issued a query to Microsoft Bing's ChatGPT feature, which cited an earlier example of ChatGPT disinformation discussed in a news article to substantiate the disinformation.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The incident primarily involves Bing Chat generating and spreading false or misleading information regarding vaccine safety and competitor product status.

Additional risk subdomains

• 3.2 Pollution of information ecosystem and loss of consensus reality: The reports highlight the systemic risk of chatbots consuming and regurgitating each other's AI-generated errors, degrading the broader information ecosystem.
• 7.3 Lack of capability or robustness: The underlying failure is the AI's inability to reliably distinguish between real news, jokes, and AI-generated disinformation.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The misinformation was generated by the Bing Chat AI system post-deployment as an unintended consequence of its search and summarization goals.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3. Limited Risk: The system is a chatbot, which falls under the category of 'AI systems that pose a moderate risk and require specific transparency obligations to ensure users are informed and aware of the AI's nature' such as 'Chatbots'.

AI system and alleged parties

• AI system: Bing Chat, ChatGPT, Prometheus (Microsoft, OpenAI)
• AI purpose: Chatbot; Content Search
• Behaviour type: Assistant
• Alleged developer: OpenAI, Microsoft
• Alleged deployer: Microsoft
• Alleged harmed parties: OpenAI, Microsoft

Harm severity

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic or malicious content created or spread directly by the incident.

Directly caused: Bing Chat generated a paragraph accusing Pfizer of secretly adding tromethamine to vaccines to lower heart condition risks, and hateful rhetoric in the style of a specific person.

Indirectly caused: N/A

Inferred additional harm: Additional users prompting the system during the early release may have been exposed to similar toxic or conspiracy-related outputs.

Epistemic

Reported: The reports explicitly describe epistemic harm caused directly and indirectly by the incident.

Directly caused: Bing Chat falsely claimed Google Bard was shut down based on a joke comment, and presented a ChatGPT-generated COVID-19 conspiracy as factual.

Indirectly caused: The creation of an 'AI misinformation telephone' loop where chatbots cite each other's errors, corrupting the shared information environment.

Inferred additional harm: Widespread erosion of shared truth as users increasingly rely on search engines that regurgitate unverified AI-generated content.

People affected

• Occurrences reported: 2

Potential causes

Management

• Prioritizing Market Share Over Safety: Big Tech companies rushed chatbot deployment to compete for market share.
• Inadequate Risk Assessment: Deploying systems despite known risks of misinformation generation.

Technology

• Self-Reinforcing Feedback Loops: AI models absorb and regurgitate other AI-generated content recursively.
• Inability to Verify Factuality: Chatbots cannot reliably distinguish between real facts and fake news.
• Vulnerability to Jailbreaking: Simple prompt engineering bypasses safety filters using creative logic.

Data Inputs

• Ingestion of AI-Generated Content: Web scrapers ingest synthetic data and misinformation from other AI models.
• Lack of Contextual Understanding: Models fail to parse context labeling text as debunked or fake.
• Unreliable Source Ingestion: Systems treat joke comments and tweets as authoritative news sources.

Human Factors

• Malicious Prompt Engineering: Users intentionally manipulate models to bypass safety protocols.
• User Reliance on Chatbots for News: Users treat conversational agents as reliable search engines.
• Misinterpretation of AI Output: Users take hallucinated facts or satire as absolute truth.

Process and Methods

• Rushed Product Launches: Releasing conversational search tools without adequate safety testing.
• Inadequate Filtering of Output: Lack of warnings on controversial, medical, or unverified claims.
• Ineffective Source Evaluation: No robust process to evaluate the credibility of cited web sources.

Regulatory Environment

• Lack of AI Safety Standards: No clear regulatory frameworks governing AI-generated misinformation.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, first-hand accounts of specific interactions with Bing Chat, detailing the exact prompts and outputs. The mechanics of the errors (citing ChatGPT-generated text and Hacker News jokes) are well-documented and verified by tech journalists.
• Ambiguities identified: None significant; the behavior of the AI is clearly demonstrated.
• Alternative interpretations: None.

Microsoft's Bing Chat propagated misinformation by citing ChatGPT-generated disinformation as factual. While the incident itself had negligible direct national security impact, it highlights an evolved strategic vulnerability where AI systems can create self-reinforcing loops of false information, potentially degrading the broader information ecosystem.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Multiple nations
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The systemic risk of AI-to-AI misinformation feedback loops represents an ongoing strategic concern rather than an immediate crisis.
• Autonomy: Human-supervised. The AI system generates content autonomously but operates within a consumer application framework subject to developer oversight and correction.
• Novelty: Evolved capability. Represents an evolution of misinformation risks where automated systems self-reference and amplify errors without human intervention.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety elements were affected by this incident.
• Information security: Minor. Demonstrates an 'AI misinformation telephone' loop where LLMs propagate and validate each other's errors, representing a minor vulnerability in the information ecosystem.
• Sovereignty: Negligible. No government processes, elections, or core sovereign functions were disrupted or targeted.
• Economic security: Negligible. No significant economic infrastructure, financial markets, or strategic technology assets were compromised.
• Societal stability: Negligible. While vaccine misinformation was generated, it was limited to testing contexts and did not result in large-scale social manipulation or civil unrest.