Thousands of Incorrect ChatGPT-Produced Answers Posted on Stack Overflow

Thousands of incorrect answers produced by OpenAI's ChatGPT were submitted to Stack Overflow, which swamped the site's volunteer-based quality curation process and harmed users looking for correct answers.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The primary issue is ChatGPT generating incorrect and misleading coding answers that look plausible, leading to the spread of false information on Stack Overflow.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The chatbot fails to perform reliably, frequently generating incorrect code and non-existent references due to limitations in its factual accuracy.
• 6.3 Economic and cultural devaluation of human effort: The flood of low-effort AI-generated content swamped and overwhelmed the volunteer-based human curation infrastructure of the platform.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was driven by ChatGPT generating inaccurate but plausible-sounding answers, which is an action of the AI system, occurring after its public deployment.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a chatbot (ChatGPT), which falls under the category of AI systems that require transparency obligations so users are aware they are interacting with an AI.

AI system and alleged parties

• AI system: ChatGPT 3.5 (OpenAI)
• AI purpose: Question Answering; Code Generation
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI
• Alleged harmed parties: Stack Overflow users, Stack Overflow

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct not assessed, indirect not assessed
• Infrastructure: direct not assessed, indirect not assessed
• Property: direct not assessed, indirect not assessed
• Financial: direct not assessed, indirect not assessed
• Environmental: direct not assessed, indirect not assessed
• Malicious content: direct not assessed, indirect not assessed
• Differential treatment: direct not assessed, indirect not assessed
• Civil rights: direct not assessed, indirect not assessed
• Democracy: direct not assessed, indirect not assessed
• Privacy: direct not assessed, indirect not assessed
• Psychological: direct not assessed, indirect not assessed
• Epistemic: direct Minor, indirect Substantial
• Child sexual exploitation and abuse: direct not assessed, indirect not assessed

Epistemic

Reported: Yes, the reports explicitly describe epistemic harm in the form of widespread incorrect information, fabricated citations, and non-existent references generated by ChatGPT.

Directly caused: ChatGPT generated plausible-sounding but incorrect answers, fake quotes from CEOs, and non-existent mathematical references.

Indirectly caused: Users posted thousands of these unverified, incorrect answers to Stack Overflow, polluting the platform's knowledge base.

Inferred additional harm: It is likely that many developers who used these AI-generated answers spent significant time debugging incorrect code or implemented flawed software solutions.

People affected

• Occurrences reported: 1
• People reportedly harmed: 100
• People reportedly exposed: 10000

Potential causes

Management

• No proactive risk assessment: Stack Overflow did not anticipate the rapid volume of AI-generated spam.
• OpenAI rapid public release: Releasing the free tool publicly without safeguards created external chaos.

Technology

• Plausible but false text generation: GPT-3.5 generates authoritative sounding but incorrect answers.
• No factual reasoning rules: LLMs predict words statistically instead of using hard-coded factual rules.
• Fabrication of citations: The model hallucinates fake references and quotes to appear authentic.

Data Inputs

• Web scraping lacks truth filters: Training on unchecked web data causes the model to mimic incorrect patterns.

Human Factors

• Unverified posting by users: Users post AI answers without verifying their correctness.
• Blind trust in AI authority: Users assume authoritative-sounding AI responses must be correct.
• Hype-driven behavior: Users blindly adopt ChatGPT to post high volumes of content.

Process and Methods

• Overwhelmed curation workflow: Volunteer review process cannot handle thousands of rapid AI submissions.
• Lack of AI detection tools: Moderators lack reliable tools to quickly identify AI-generated content.

Regulatory Environment

• Lack of platform AI policies: No pre-existing standards existed to regulate AI-generated content.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from Stack Overflow moderators and tech media provide clear, consistent, and detailed accounts of the incident, the volume of posts, and the reasons for the temporary ban. There is little ambiguity about what occurred.

Stack Overflow's temporary ban on ChatGPT-generated content highlighted the challenges of plausible but incorrect AI-generated information at scale. While it disrupted a major software development community, the incident had negligible national security implications, representing a platform-level moderation challenge rather than a threat to national infrastructure, sovereignty, or security.

• Overall national security impact: Negligible
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global developer community
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The proliferation of LLM-generated misinformation represents an ongoing strategic and capability challenge rather than an immediate national security crisis.
• Autonomy: Human-controlled. The AI tool was prompted by individual users who chose to copy and post the generated content, maintaining human control over the final action.
• Novelty: Evolved capability. Represented a significant advancement in the scale and plausibility of automated text generation, building on existing spam and bot capabilities.

Impact by dimension

• Physical security: Negligible. The incident involved incorrect coding answers on a public Q&A website and did not impact physical systems, critical infrastructure, or human safety.
• Information security: Negligible. While the incident involved the spread of incorrect information, it was driven by uncoordinated public users rather than a state-sponsored disinformation or intelligence operation.
• Sovereignty: Negligible. No impact on state authority, government operations, electoral systems, or constitutional processes.
• Economic security: Negligible. The disruption was limited to a developer forum's moderation system and did not threaten strategic industries, national financial systems, or core economic security.
• Societal stability: Negligible. The incident did not involve mass surveillance, systematic discrimination, or threats to civil liberties and social cohesion.