IBM Watson for Oncology Criticized by Customers for Allegedly Unsafe and Inaccurate Cancer Treatment Recommendations

Internal documents from IBM Watson Health showed negative assessments from customers such as Florida’s Jupiter Hospital and Memorial Sloan Kettering criticizing its Watson for Oncology product for allegedly unsafe and incorrect cancer treatment recommendations.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The Watson for Oncology system failed to perform reliably, generating inaccurate and potentially dangerous treatment recommendations that deviated from standard medical guidelines.

Additional risk subdomains

• 7.4 Lack of transparency or interpretability: IBM was not transparent with users about the source of Watson's training data, falsely claiming it was based on real patient data and broad consensus.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The unsafe and inaccurate treatment recommendations were generated by the Watson for Oncology AI system after its deployment to hospitals worldwide.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is an AI tool used in healthcare for diagnostic and treatment recommendations. According to the definitions, High Risk includes 'AI used in healthcare, such as diagnostic tools.'

AI system and alleged parties

• AI system: Watson for Oncology (IBM)
• AI purpose: Medical Diagnosis Support; Health AI Assistant
• Behaviour type: Assistant
• Alleged developer: IBM Watson Health
• Alleged deployer: Memorial Sloan Kettering, Jupiter Hospital
• Alleged harmed parties: oncologists, cancer patients

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Epistemic

Reported: The report explicitly describes epistemic harm in the form of the AI generating incorrect and unsafe medical recommendations, and IBM presenting misleading information about the system's training data and study results.

Directly caused: The Watson system directly generated erroneous cancer treatment recommendations, such as advising a dangerous drug combination for a bleeding lung cancer patient.

Indirectly caused: IBM's misleading marketing and biased concordance studies misled physicians and hospitals about the true capabilities and evidence base of the AI system.

Inferred additional harm: The dissemination of inaccurate medical advice could have eroded trust in AI-driven clinical decision support systems across the medical community.

People affected

• Occurrences reported: 1
• People reportedly exposed: 1

Potential causes

Management

• Prioritization of Marketing: Executives promoted the tool for marketing despite knowing of systemic flaws.
• Misleading Public Claims: Management falsely claimed the system was trained on real patient data.
• Suppression of Internal Warnings: Executives ignored internal reports detailing unsafe medical recommendations.

Technology

• Slow adaptation to new research: The software was slow to adapt to changing medical guidelines and research.
• Inaccurate Recommendation Algorithm: The algorithm generated unsafe treatment options contrary to drug warnings.

Data Inputs

• Use of Synthetic Training Data: Watson was trained on hypothetical cases instead of real-world patient data.
• Small and Biased Dataset: Training used a small number of cases chosen without statistical input.
• Narrow Expert Input: Training relied on the preferences of only one or two doctors per cancer.

Human Factors

• Overreliance on Limited Expertise: Engineers relied on a few specialists instead of broad clinical guidelines.
• Unconventional Clinical Judgement: Training doctors injected preferences that deviated from standard guidelines.

Process and Methods

• Flawed Concordance Study Design: Validation studies were designed to make negative findings highly unlikely.
• Lack of Statistical Validation: The training case sizes were determined without rigorous statistical input.

Regulatory Environment

• Lack of Strict Medical AI Oversight: Absence of regulatory checks allowed unsafe recommendations to be marketed.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide detailed, consistent information from internal IBM documents, customer feedback, and expert opinions regarding the system's training flaws and incorrect recommendations. There is clear consensus on the technical issues (synthetic data, small sample size) and the lack of reported physical harm to actual patients.
• Ambiguities identified: The exact number of hospitals actively using the flawed recommendations on real patients versus system testing is slightly unclear, though the report states 230 hospitals worldwide use it.
• Alternative interpretations: None. The failure of the AI system to provide safe recommendations due to poor training data is unambiguous.

IBM Watson for Oncology provided unsafe and inaccurate cancer treatment recommendations due to being trained on synthetic, non-real-world patient data. While the incident caused significant reputational and financial damage to IBM and highlighted critical safety verification issues in medical AI, its direct national security impact is minor and restricted to technological competitiveness concerns.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global
• Alleged perpetrator: IBM

Threat characteristics

• Imminence: Long-term. This represents an ongoing strategic concern regarding the safety, validation, and marketing of clinical AI decision support systems rather than an active crisis.
• Autonomy: Human-supervised. The AI operated as a clinical decision support tool where human physicians retained final decision-making authority and oversight, preventing patient harm.
• Novelty: Evolved capability. Represents an evolved threat where advanced cognitive computing systems generate complex, plausible-sounding but clinically unsafe recommendations based on flawed synthetic training data.

Impact by dimension

• Physical security: Negligible. The AI provided unsafe treatment recommendations, but human physicians intercepted them, resulting in zero reported physical harm. Standard medical oversight mitigated any physical risk.
• Information security: Negligible. No national intelligence, classified systems, or systematic information warfare operations were involved in this commercial healthcare AI failure.
• Sovereignty: Negligible. The incident involved commercial medical software and did not impact state authority, border control, or core government operations.
• Economic security: Minor. Represents minor technological security impact through reputational damage to a major US technology firm (IBM) in the global healthcare AI market and financial losses for adopting hospitals.
• Societal stability: Negligible. No evidence of mass surveillance, systemic civil rights violations, or threats to societal stability, though it highlights general challenges in medical AI trust.