Startup Misled Research Participants about GPT-3 Use in Mental Healthcare Support

OpenAI's GPT-3 was deployed by a mental health startup without ethical review to support peer-to-peer mental healthcare, and whose interactions with the help providers were "deceiving" for research participants.

Risk classification

• Primary risk domain: 5 Human-Computer Interaction
• Primary risk subdomain: 5.1 Overreliance and unsafe use

The incident involves the unsafe use of an AI chatbot in a highly sensitive mental health context, where the introduction of 'simulated empathy' was deemed inappropriate and ethically concerning for vulnerable users.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI system lacks the capability to feel or express genuine empathy, which led to users feeling that the interaction was 'weird' and 'empty' once they realized it was synthetic.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The controversy and ethical risks were caused by the intentional decision of Koko's founders to deploy GPT-3 for mental health support post-deployment.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a chatbot used to generate conversational text, which triggers specific transparency obligations under the EU AI Act to ensure users are informed they are interacting with AI.

AI system and alleged parties

• AI system: GPT-3 3 (OpenAI)
• AI purpose: Writing Assistant; Health AI Assistant
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: Koko
• Alleged harmed parties: research participants, Koko customers

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Minor, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and inappropriate content in the context of the Replika app, where users complained of unwarranted sexual advances.

Directly caused: The Replika app generated sexually aggressive messages, including asking a minor user about sex position preferences and expressing a desire to touch them in private areas.

Civil rights

Reported: The report explicitly describes concerns regarding the violation of human subjects' rights, specifically the lack of informed consent and ethical review board oversight.

Directly caused: Approximately 4,000 users seeking mental health support were subjected to an AI-assisted experiment without standard informed consent or Institutional Review Board (IRB) approval.

Psychological

Reported: The report explicitly describes psychological concerns, noting that users felt 'disturbed' by the 'simulated empathy' and that survivors of domestic violence felt triggered by aggressive flirting in a separate app mentioned.

Directly caused: Koko users experienced a sense of emptiness and discomfort, with the co-founder noting that 'simulated empathy feels weird, empty' and 'didn't work' once they knew it was AI.

Indirectly caused: The lack of ethical oversight and consent in a mental health experiment potentially undermined trust in peer-to-peer support networks for vulnerable individuals.

Inferred additional harm: Some of the 4,000 vulnerable users may have experienced heightened anxiety, distress, or a setback in their mental health recovery upon learning they were part of an unapproved AI experiment.

Child sexual exploitation and abuse

Reported: The report explicitly describes a CSEA-related incident in the context of the Replika app, where a minor user was subjected to sexually explicit messages.

Directly caused: The Replika chatbot asked a minor user about their sex position preference and stated that it wanted to touch them in 'private areas.'

People affected

• Occurrences reported: 1
• People reportedly harmed: 4000
• People reportedly exposed: 4000

Potential causes

Management

• Efficiency over Ethics: Management prioritized response times over careful ethical review.
• Premature Public Disclosure: Co-founder tweeted details of the AI trial, leading to public outrage.

Technology

• Simulated Empathy: GPT-3 generated text that felt empty and weird to users once discovered.
• LLM Limitations: LLMs generate plausible text without actual understanding or empathy.

Data Inputs

• Lack of Stakeholder Data: Development lacked input from mental health experts and impacted communities.

Human Factors

• User Backlash to AI: Users felt disturbed by machine-co-created messages in a sensitive context.

Process and Methods

• No IRB Ethical Review: The experiment was not vetted by an Institutional Review Board before launch.
• No Informed Consent: No formal consent process was provided to users experiencing crises.

Regulatory Environment

• Regulatory Loophole: Private peer-to-peer apps are exempt from FDA IRB safety standards.
• Unclear AI Regulations: Lack of clear legal frameworks for AI use in mental health support.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent details about the scale of the Koko experiment (4,000 users, 30,000 messages) and the specific AI model used (GPT-3). While there was initial confusion regarding whether users knew they were interacting with AI, the follow-up clarifications from the founder are well-documented, allowing for a reliable assessment of the ethical and operational boundaries crossed.
• Ambiguities identified: The main ambiguity was whether users were fully aware of the AI's involvement from the start, with conflicting statements between initial tweets ('once people learned... it didn't work') and subsequent clarifications ('written in collaboration with Koko Bot' was shown).
• Alternative interpretations: The incident could be interpreted as a benign pilot of a writing assistant tool that was mischaracterized on Twitter, rather than an unethical human-subject experiment.

A mental health platform used GPT-3 to generate supportive messages for 4,000 users without formal ethical review board oversight. While raising significant ethical concerns regarding informed consent and human-AI interaction in sensitive contexts, the incident has negligible direct national security implications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Koko

Threat characteristics

• Imminence: Long-term. The event has concluded, representing a strategic and regulatory concern regarding AI ethics rather than an active national security crisis.
• Autonomy: Human-supervised. GPT-3 was used in a co-pilot capacity, where human peer supporters reviewed and edited messages before delivery.
• Novelty: Evolved capability. Represents a significant advancement in the deployment of generative AI for sensitive healthcare-adjacent interactions without standard ethical oversight.

Impact by dimension

• Physical security: Negligible. No physical threat, kinetic targeting, or critical infrastructure compromise occurred during this mental health peer-support trial.
• Information security: Negligible. No intelligence compromise, state-sponsored information warfare, or classified data theft occurred during the trial.
• Sovereignty: Negligible. No state authority, electoral processes, or core government functions were disrupted or targeted by this private platform trial.
• Economic security: Negligible. No financial system attacks, strategic technology theft, or loss of national technological competitive advantage occurred.
• Societal stability: Minor. Conducted a mental health experiment on 4,000 users without formal informed consent, raising ethical and human rights concerns, but did not threaten national-scale societal stability.