Epic Systems’s Sepsis Prediction Algorithms Revealed to Have High Error Rates on Seriously Ill Patients

Epic System's sepsis prediction algorithms was shown by investigators at the University of Michigan Hospital to have high rates of false positives and false negatives, allegedly delivering inaccurate and irrelevant information on patients, contrasting sharply with their published claims.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The Epic Sepsis Model failed to perform robustly, missing 67% of sepsis cases and generating 88% false alarms in an independent validation study, demonstrating a severe lack of capability in a critical clinical application.

Additional risk subdomains

• 7.4 Lack of transparency or interpretability: The ESM's mathematical formulas and training data were proprietary and shielded from public scrutiny by a corporate firewall.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The underperformance and false alerts of the Epic Sepsis Model occurred post-deployment, representing an unintentional failure of the AI system to accurately predict sepsis.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is an AI tool used in healthcare for diagnostic support ('medical AI systems' and 'tools that aid physicians in treatment and diagnosis'), which has significant implications for patient safety and health outcomes.

AI system and alleged parties

• AI system: Epic Sepsis Model (Epic Systems)
• AI purpose: Medical Diagnosis Support; Health AI Assistant
• Behaviour type: Assistant
• Alleged developer: Epic Systems
• Alleged deployer: University of Michigan Hospital
• Alleged harmed parties: sepsis patients

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Psychological

Reported: The reports explicitly mention that the high rate of false alarms (88%) created a 'large burden of alert fatigue' for clinicians.

Directly caused: The model generated 6,128 false alarms out of 6,971 alerts, causing significant alert fatigue and cognitive stress for clinicians at Michigan Medicine.

Inferred additional harm: Alert fatigue likely caused increased anxiety and stress among clinical staff across the 170 hospitals using the system, potentially leading to burnout or clinical errors.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1709
• People reportedly exposed: 38455

Potential causes

Management

• Business Goals Over Safety: Preserving market dominance clashed with the need for independent review.
• Financial Conflicts of Interest: Vendor paid hospitals up to $1M to use algorithms, skewing safety priorities.
• Lack of Developer Transparency: Proprietary models were shielded from scrutiny by a corporate firewall.

Technology

• Proprietary Black Box Algorithms: Details inside the 'black boxes' are secret and independent tests are scarce.
• High False Positive Rates: The model generated 88% false alarms, creating severe alert fatigue.
• Algorithm Concept Drift: Nonadaptive algorithms perform worse over time due to concept drift.

Data Inputs

• Biased Training Data: Algorithms trained on past spending data which reflects historical racial bias.
• Geographic Data Bias: Training data relied on few states, causing poor performance elsewhere.
• Inconsistent Data Terminology: Different hospitals used different terminology to catalog medical records.

Human Factors

• Alert Fatigue: Clinicians faced thousands of false alerts, causing them to ignore warnings.
• Uncritical Trust in AI: Belief in 'AI hype' led hospitals to trust algorithms without independent tests.
• Failure to Calibrate: Hospitals failed to calibrate and tune the model for their local population.

Process and Methods

• Lack of Local Validation: Hospitals deployed the tool without testing it in their clinical environment.
• Varying Disease Definitions: Sepsis definitions varied, causing discrepancies between model and practice.
• No Postmarket Surveillance: No routine monitoring was in place to track safety and performance over time.

Regulatory Environment

• Lack of FDA Oversight: Many clinical algorithmic tools were not required to be reviewed by the FDA.
• No Mandatory Reporting: No central reporting system exists for patient harm caused by faulty AI.
• No Required Bias Testing: Testing for racial and ethnic bias is suggested but not legally required.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide highly detailed, peer-reviewed scientific data from a JAMA Internal Medicine study validating the model's performance, including specific metrics like sensitivity (33%), specificity (83%), and false alarm rates (88%). The roles of the AI developer (Epic) and the validating institution (Michigan Medicine) are clearly defined, leaving very little ambiguity about the system's performance and the nature of the incident.
• Ambiguities identified: There is some ambiguity regarding whether any specific patient deaths or injuries can be directly traced to the ESM's missed alerts, as the study was a retrospective analysis of records.
• Alternative interpretations: Epic argued that the poor performance was due to a lack of local calibration by the hospital rather than an inherent flaw in the model itself.

An independent study revealed that the widely deployed Epic Sepsis Model missed 67% of sepsis cases and caused severe alert fatigue due to an 88% false-alarm rate. While representing a significant patient safety and clinical quality concern within the critical infrastructure of healthcare, the national security implications remain minor and manageable within standard healthcare regulatory and operational frameworks.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents a systemic, ongoing issue with clinical AI validation rather than an active national security crisis requiring immediate response.
• Autonomy: Human-controlled. The AI model functions as a decision-support tool, generating alerts while human clinicians retain final decision-making authority over patient care.
• Novelty: Established threat. Software and algorithmic failures in clinical decision support systems are an established risk within the healthcare sector.

Impact by dimension

• Physical security: Minor. Healthcare is a critical infrastructure sector. Widespread deployment of the faulty model across hundreds of hospitals degraded clinical operations through severe alert fatigue and missed diagnoses, but the impact is manageable within standard clinical safety and quality procedures.
• Information security: Negligible. The incident involves a clinical diagnostic tool and does not impact intelligence collection, classified systems, or information operations.
• Sovereignty: Negligible. No impact on state authority, territorial integrity, or core government decision-making processes.
• Economic security: Negligible. While hospitals faced potential financial losses from integration and calibration, this does not threaten national economic stability or strategic technological advantages.
• Societal stability: Negligible. The failure of the diagnostic tool does not involve mass surveillance, systemic discrimination, or threats to civil liberties.