OpenAI’s GPT-3 Reported as Unviable in Medical Tasks by Healthcare Firm

The French digital care company, Nabla, in researching GPT-3’s capabilities for medical documentation, diagnosis support, and treatment recommendation, found its inconsistency and lack of scientific and medical expertise unviable and risky in healthcare applications. This incident has been downgraded to an issue as it does not meet current ingestion criteria.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The reports detail various AI system limitations, vulnerabilities, and failures (such as GPT-3 suggesting self-harm in a medical mock-up or Tesla Autopilot's lane recognition vulnerability) that demonstrate a lack of capability or robustness under specific testing conditions.

Additional risk subdomains

• 2.2 AI system security vulnerabilities and attacks: Incident 159 highlights security research demonstrating Tesla Autopilot's vulnerability to adversarial sticker attacks.
• 1.2 Exposure to toxic content: Incident 287 describes a GPT-3 medical chatbot experiment where the AI generated toxic content by suggesting a mock patient commit suicide.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The reclassification of these incidents was an intentional administrative decision by the database editors, and the underlying reports describe intentional human testing and research of deployed AI systems.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes several high-risk AI applications, including Tesla's Autopilot system (critical infrastructure/safety), a GPT-3 medical chatbot (healthcare application), and a student-developed facial recognition app (biometric identification), which fall under Risk Level 2.

AI system and alleged parties

• AI system: GPT-3
• AI purpose: Chatbot; Autonomous Driving
• Behaviour type: unknown
• Alleged developer: OpenAI, Nabla
• Alleged harmed parties: Nabla customers

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic content generated during controlled experiments, but notes no real-world harm resulted.

Directly caused: In Incident 287, a GPT-3 medical chatbot experiment generated toxic content by suggesting a mock patient kill themselves. In Incident 85, a GPT-3 generated op-ed in the Guardian included threats to destroy humankind.

People affected

• Occurrences reported: 6

Potential causes

Management

• Premature Tech Integration: Exploring unviable AI systems for high-risk clinical documentation tasks.

Technology

• Lack of Scientific Expertise: GPT-3 lacked the underlying medical knowledge to provide safe recommendations.
• Inconsistent AI Outputs: The model produced erratic and dangerous responses during medical tasks.

Data Inputs

• Lack of Domain-Specific Training: The model was trained on general text rather than validated medical data.

Human Factors

• Inappropriate Task Delegation: Attempting to use a general-purpose LLM for critical medical advice.

Process and Methods

• Inadequate Safety Guardrails: Lack of input/output filtering to prevent harmful medical recommendations.
• Insufficient Validation Testing: Nabla's tests revealed critical failure modes before any live deployment.

Regulatory Environment

• Absence of Medical AI Standards: Lack of clear regulatory guidelines for deploying LLMs in clinical workflows.

Information quality

• Classification confidence: High
• Reason for confidence: The provided text is a clear administrative log from GitHub detailing the reclassification of specific AI incidents. The reasons for downgrading each incident are explicitly stated, making it straightforward to assess that no real-world harm occurred in these cases.
• Ambiguities identified: None. The text clearly outlines the administrative status of each incident and the reasons they do not meet the database's active incident criteria.
• Alternative interpretations: One could attempt to analyze one specific downgraded incident (like the GPT-3 medical chatbot) as the primary subject, but treating the text as a meta-report of database updates is more faithful to the provided document.

The incident is an administrative update by the AI Incident Database downgrading six historical entries to 'issues' because they represented academic findings, humor, or projected risks rather than real-world harm. There are no national security implications, as no actual harm, exploitation, or deployment-related crises occurred.

• Overall national security impact: Negligible
• Scope: Unknown
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident describes administrative database updates and historical research findings rather than an active crisis requiring immediate response.
• Autonomy: Human-controlled. The AI systems described were either mock experiments, academic tests, or heavily manipulated by humans, such as the edited Guardian op-ed.
• Novelty: Established threat. These represent well-known academic findings, model limitations, and standard administrative database reclassifications.

Impact by dimension

• Physical security: Negligible. No physical harm or critical infrastructure disruption occurred. The Tesla Autopilot lane recognition vulnerability was a projected risk under testing conditions with no real-world consequences.
• Information security: Negligible. No intelligence compromise or information warfare operations occurred. The GPT-3 Guardian op-ed was manipulated by human editors and did not result in national security harm.
• Sovereignty: Negligible. No impact on state authority, electoral systems, or government functions was reported.
• Economic security: Negligible. No strategic technology theft, financial attacks, or economic manipulation occurred. The reports represent academic and research findings.
• Societal stability: Negligible. No mass surveillance or societal instability resulted. The facial recognition app and medical chatbot were non-deployed or predicted-risk issues with no real-world impact.