ChatGPT Reportedly Misleads Users About Soundslice Features, Allegedly Prompting Unplanned Product Development

ChatGPT reportedly misinformed users that Soundslice's music software could import ASCII tablature, a feature the company did not offer. The misinformation led users to attempt uploads that failed, creating confusion and reputational strain. Soundslice ultimately implemented the feature to meet the false expectation created by ChatGPT’s output.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

ChatGPT generated false information claiming Soundslice supported ASCII tab imports, misleading users and forcing the company to adapt.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The incident stems from ChatGPT's technical limitation of hallucinating false facts, representing a lack of capability or robustness in generating accurate information.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by ChatGPT hallucinating a feature post-deployment, which was an unintentional outcome of its text generation capabilities.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a general-purpose chatbot (ChatGPT), which falls under Limited Risk due to transparency obligations to ensure users know they are interacting with an AI.

AI system and alleged parties

• AI system: ChatGPT (OpenAI)
• AI purpose: Chatbot; Question Answering
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI
• Alleged harmed parties: SoundSlice

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 100

Potential causes

Management

• Rapid Deployment of LLMs: Releasing conversational AI without robust accuracy safeguards.

Technology

• LLM Hallucination: ChatGPT generated false facts about Soundslice capabilities.
• Lack of Real-Time Verification: Model could not verify third-party features dynamically.

Data Inputs

• Outdated Training Data: Training data lacked precise current features of third-party apps.
• Unstructured Web Association: Model falsely linked Soundslice to ASCII tab processing.

Human Factors

• User Overreliance on AI: Users blindly trusted ChatGPT instructions without verification.

Process and Methods

• Inadequate Output Filtering: OpenAI lacked filters to catch false claims about external tools.

Regulatory Environment

• Lack of AI Liability Rules: No regulations hold AI creators liable for business disruptions.

Information quality

• Classification confidence: High
• Reason for confidence: The report is a first-hand account from the founder of Soundslice, detailing the exact nature of the AI's hallucination, the evidence found in error logs, and the company's response. There is little ambiguity about what transpired.

An OpenAI ChatGPT hallucination falsely claimed that the music software Soundslice supported ASCII tab imports. This led users to upload incompatible files, straining the company's reputation and forcing them to build an unplanned feature. The incident represents a novel business impact of AI hallucinations but carries negligible national security implications.

• Overall national security impact: Negligible
• Scope: Unknown
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This represents an ongoing strategic concern regarding LLM reliability rather than an active security crisis.
• Autonomy: Human-controlled. The AI acts as an assistant providing information, while human users make the final decision to act on its suggestions.
• Novelty: Evolved capability. While LLM hallucinations are established, this represents an evolved impact where hallucinated capabilities directly dictate a private company's product roadmap.

Impact by dimension

• Physical security: Negligible. The incident involves a music software platform and has no connection to physical systems, critical infrastructure, or human safety.
• Information security: Negligible. While the incident involves AI-generated misinformation, it is a benign technical error regarding software features rather than a coordinated information warfare campaign.
• Sovereignty: Negligible. No impact on government operations, sovereignty, or state authority.
• Economic security: Negligible. The economic impact is limited to minor resource redirection for a single commercial music software company, presenting no threat to national economic or technological security.
• Societal stability: Negligible. The incident does not affect societal stability, civil liberties, or human rights.