ChatGPT Account Compromise Leads to Unintended Data Exposure

A security breach involving ChatGPT led to the exposure of sensitive conversations, including login credentials and personal data, after a user account was compromised. OpenAI responded to the incident with an explanation.

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information

The incident involved the exposure of non-public sensitive information, including pharmacy portal login credentials and unpublished research details, to an unrelated user.

Additional risk subdomains

• 2.2 AI system security vulnerabilities and attacks: The exposure was facilitated by an account takeover and a lack of standard security features like 2FA on the ChatGPT platform.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was primarily caused by human actions, specifically an intentional account takeover by external actors and a lack of standard security features implemented by OpenAI developers.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: ChatGPT is a general-purpose chatbot, which falls under the category of AI systems that pose a moderate risk and require specific transparency obligations.

AI system and alleged parties

• AI system: ChatGPT (OpenAI)
• AI purpose: Chatbot; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI
• Alleged harmed parties: ChatGPT users, Chase Whiteside

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Privacy

Reported: Yes, the report explicitly describes privacy violations where private chat histories containing login credentials and personal details were exposed.

Directly caused: Private chat histories of multiple unrelated users, including pharmacy portal credentials, an unpublished research proposal, and a PHP script, were exposed to an unrelated user.

Inferred additional harm: It is likely that other users whose accounts were part of the credential pool also had their private chat histories exposed to unauthorized parties.

People affected

• Occurrences reported: 1
• People reportedly harmed: 5
• People reportedly exposed: 5

Potential causes

Management

• Prioritizing Speed Over Security: Rapid deployment of features without standard security controls like 2FA.
• Inadequate User Access Controls: Management failure to mandate industry-standard authentication methods.

Technology

• Lack of 2FA: No multi-factor authentication allowed unauthorized access to user accounts.
• No Session IP Tracking: Users could not monitor active sessions or login locations to detect breaches.
• Potential Caching Glitches: Middlebox caching errors could map active sessions to unrelated users.

Data Inputs

• Input of Sensitive Credentials: Users inputting sensitive system credentials into the chatbot interface.
• Proprietary Data Sharing: Users entering proprietary code and presentation details into queries.
• Lack of Input Sanitization: System accepted plaintext credentials and sensitive data without warnings.

Human Factors

• Credential Sharing or Reuse: Users reusing passwords across multiple accounts or platforms.
• Overreliance on AI Privacy: Users assuming conversations with the AI chatbot are completely private.
• Poor Security Awareness: Employees troubleshooting secure portals using public AI tools.

Process and Methods

• Weak Account Security Policies: Absence of basic security standards like login alerts or 2FA.
• Inadequate Session Management: Lack of robust validation for concurrent or geographically distant sessions.
• Inadequate Data Leak Prevention: Lack of client-side checks to prevent pasting sensitive information.

Regulatory Environment

• Lack of AI Data Standards: Absence of regulatory frameworks governing data privacy in LLM applications.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear details of the incident, including the user's account, the nature of the leaked data, and OpenAI's official explanation of an account takeover. There is minor disagreement between the user and OpenAI regarding whether the account was compromised, but the overall mechanism of exposure is well-explained.
• Ambiguities identified: There is a slight conflict between the user's belief that his account was not compromised and OpenAI's finding of successful logins from Sri Lanka.
• Alternative interpretations: The incident could be interpreted as a direct system bug leaking histories (as initially suspected) rather than an account takeover, though OpenAI's investigation strongly points to the latter.

Consumer ChatGPT accounts were compromised via credential stuffing originating from Sri Lanka, leading to the exposure of private chat histories, including commercial credentials and unpublished research. The incident represents a minor privacy and commercial security issue rather than a national security threat, manageable through standard cyber hygiene and platform security updates.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Sri Lanka, United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The immediate incident is resolved, but it highlights ongoing, long-term strategic concerns regarding data privacy and security hygiene in consumer AI platforms.
• Autonomy: Human-controlled. The AI system functions as a passive assistant, and the security compromise resulted entirely from human actions such as credential stuffing and account takeover.
• Novelty: Established threat. Credential stuffing, account takeovers, and unauthorized data exposure are highly common and established cyber security threats, not a novel AI-specific exploit.

Impact by dimension

• Physical security: Negligible. No kinetic threats, weapons deployment, or critical infrastructure manipulation occurred. The leaked pharmacy portal credentials represent a commercial/privacy issue rather than a national security threat to critical infrastructure.
• Information security: Negligible. No intelligence capabilities or classified data were compromised. The leaked research proposal and PHP script do not constitute state-level intelligence or information warfare.
• Sovereignty: Negligible. Core government operations, electoral systems, and state sovereignty were entirely unaffected by this consumer-level account compromise.
• Economic security: Minor. Minor risk to intellectual property (unpublished research proposal) and commercial credentials. Manageable through standard security procedures like password resets and implementing multi-factor authentication.
• Societal stability: Minor. Limited privacy compromise affecting a small number of individuals whose chat histories were exposed. Does not represent large-scale societal instability or systemic human rights violations.