COVID-19 Detection and Prognostication Models Allegedly Flagged for Methodological Flaws and Underlying Biases

Peer-review of papers about COVID-19 detection and prognostication algorithms from 2020, including deployed models, revealed none to be ready for clinical use, due to methodological flaws and underlying biases such as lacking external validation or not specifying data sources and model training details.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The incident involves hundreds of AI models that fail to perform reliably or generalize to real-world data, representing a severe lack of capability and robustness in clinical applications.

Additional risk subdomains

• 1.3 Unequal performance across groups: The reports highlight that most models failed to test or report performance across different demographic groups, risking unequal outcomes.
• 7.4 Lack of transparency or interpretability: Many studies and FDA-approved products failed to disclose training approaches, statistical methods, or validation data sources.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk of worsening patient care arises from the unintentional failure of AI models to perform reliably in real-world clinical settings due to methodological flaws.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The reports describe 'AI used in healthcare, such as diagnostic tools' which is explicitly classified as High Risk due to its significant implications for patient safety and health outcomes.

AI system and alleged parties

• AI system: None named
• AI purpose: Medical Diagnosis Support; Image Classification
• Behaviour type: Assistant
• Alleged developer: unknown, Icahn School of Medicine researchers
• Alleged deployer: unknown, Mount Sinai Hospital
• Alleged harmed parties: COVID-19 patients, COVID-19 healthcare providers

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Substantial, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Epistemic

Reported: The report explicitly describes a 'credibility crisis' and a 'polluted area of research' due to hundreds of flawed, non-reproducible scientific papers.

Directly caused: The publication of over 400 fatally flawed AI models for COVID-19 created a highly polluted scientific literature, making it difficult for researchers to trust published claims.

Inferred additional harm: The flood of low-quality, non-reproducible papers likely eroded public and professional trust in medical AI research and delayed the development of genuinely useful clinical tools.

People affected

• Occurrences reported: 1

Potential causes

Management

• Speed Prioritized Over Rigor: Pandemic pressure and competition led to rushed, flawed experiments.
• Lack of Team Collaboration: Siloed development without close collaboration between doctors and AI teams.
• Insufficient Risk Assessment: Organizations failed to evaluate clinical risks of deploying unverified AI.

Technology

• Flawed Model Generalizability: Models trained on static data fail in dynamic real-world environments.
• Data Leakage in Evaluation: Reusing the same data for training and testing inflated accuracy results.
• Algorithmic Bias: Models trained on adults were improperly tested on pediatric X-rays.

Data Inputs

• Small and Single-Origin Datasets: Limited data samples lacked geographical and demographic diversity.
• Poor Data Labeling and Quality: Datasets included unconfirmed COVID-19 cases and mislabeled images.
• Lack of Shared Datasets: Confidentiality barriers restricted access to critical validation datasets.

Human Factors

• Overreliance on Preprint Claims: Researchers trusted published claims without verifying underlying methodology.
• Reviewer Competency Gaps: Peer reviewers lacked necessary machine learning expertise to spot flaws.
• Institutional Bias: Reviewers favored prominent institutions, leading to superficial evaluations.

Process and Methods

• Lack of External Validation: Models were rarely tested on independent, real-world datasets.
• Reproducibility Failures: Underlying code and datasets were seldom disclosed for replication.
• Absence of Evaluation Standards: Lack of consensus standards or checklists like CLAIM during model review.

Regulatory Environment

• Static Regulatory Paradigm: One-time approval systems failed to account for model drift over time.
• Inadequate FDA Disclosure Rules: Approved devices lacked public disclosure of validation demographics.
• Lack of Standardized Oversight: No regulatory enforcement of methodological checklists before clinical use.

Information quality

• Classification confidence: High
• Reason for confidence: The assessment is based on a comprehensive systematic review of over 400 studies published in Nature, providing highly credible and consistent scientific consensus on the flaws of these models.
• Ambiguities identified: The exact number of patients clinically evaluated by the flawed FDA-approved models is not specified in the reports.
• Alternative interpretations: None. The evidence clearly points to a widespread failure of model robustness and scientific methodology.

A systematic review of over 400 COVID-19 AI models revealed widespread methodological flaws, presenting a 'credibility crisis' in medical machine learning. While posing potential risks to healthcare delivery and highlighting regulatory gaps in FDA approvals, the national security impact remains minor and manageable within existing health and regulatory frameworks.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This is an ongoing systemic issue regarding research quality and regulatory standards rather than an active, fast-moving crisis.
• Autonomy: Human-supervised. These AI models are designed as clinical decision support tools meant to be reviewed by medical professionals, not fully autonomous decision-makers.
• Novelty: Evolved capability. Represents a significant scaling of machine learning applications in a global health crisis, though methodological flaws in research are an established threat.

Impact by dimension

• Physical security: Minor. Widespread methodological flaws in clinical AI models pose a potential risk to patient care and healthcare system resilience during a pandemic, but impacts are currently manageable within standard health regulatory frameworks.
• Information security: Negligible. No evidence of intelligence compromise, classified data theft, or state-sponsored information warfare in the provided text.
• Sovereignty: Minor. While the FDA approved some products with limited validation data, this represents a regulatory challenge rather than a threat to state authority, territorial control, or core constitutional processes.
• Economic security: Minor. Widespread scientific flaws in medical AI research created a 'credibility crisis' in medical machine learning, potentially slowing down genuine technological advancement and eroding trust in medical AI sectors.
• Societal stability: Minor. Lack of demographic and racial validation data in 154 out of 161 FDA-approved models risks exacerbating healthcare inequalities and systemic differential treatment, though it has not triggered large-scale civil unrest.