Purportedly AI-Generated Sepsis Alert Reportedly Prompted Potentially Inappropriate IV Fluid Administration for a Dialysis Patient, Averted by Clinician Intervention

A nurse at St. Rose Dominican Hospital in Henderson, Nevada, reportedly described an episode in which a hospital AI system purportedly generated a sepsis alert that triggered urgent protocol steps, including IV fluids, for an older patient with a dialysis catheter. Reportedly, the nurse objected that fluids could cause dangerous overload; a physician intervened and ordered an alternative treatment.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI system failed to perform robustly by generating an inappropriate and dangerous clinical recommendation for a patient with a dialysis catheter, reflecting a lack of capability in handling specific patient contexts.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The charge nurse insisted on administering the fluids despite clinical objections, demonstrating overreliance on the automated AI alert over human clinical judgment.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk was initiated by an inappropriate sepsis alert generated by the deployed AI system, which did not account for the patient's specific clinical context of dialysis.

EU AI Act risk tier

• Risk tier: 2 High Risk

Risk Level 2: High Risk. The system is an AI used in healthcare for clinical risk prediction, specifically a 'sepsis-prediction algorithm' used to flag life-threatening infections, which directly impacts patient safety and clinical decision-making.

AI system and alleged parties

• AI system: hospital's artificial-intelligence system
• AI purpose: Medical Diagnosis Support; Threat Detection
• Behaviour type: Assistant
• Alleged developer: Unknown sepsis alert model developer, Unknown healthcare technology
• Alleged deployer: St. Rose Dominican Hospital (Henderson, Nevada)
• Alleged harmed parties: St. Rose Dominican Hospital (Henderson, Nevada), patients, Nurses, Epistemic integrity, Doctors

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly exposed: 1

Potential causes

Management

• Prioritizing Tech Over Staffing: Hospitals invested heavily in AI while bedside units remained understaffed.
• Top-down Implementation: Management rolled out tools without clinician input, creating mistrust.
• Inadequate Risk Assessment: Leadership failed to anticipate the operational risks of AI alert false alarms.

Technology

• Inaccurate Sepsis Algorithm: Epic sepsis model was found to be less accurate than marketed.
• Vague BioButton Alerts: BioButton generated vague alerts without specific parameters, causing confusion.
• Opaque Decision Making: AI models present scores without explaining the specific signals.

Data Inputs

• Incomplete Patient Context: Algorithms lack access to physical signs observed directly by bedside nurses.
• Biased Training Data: Some deployed algorithms were found to exhibit systemic racial bias.
• Missing Clinical Parameters: BioButton tracked vitals but missed stroke indicators like slurred speech.

Human Factors

• Alert Fatigue: High rates of false positives pulled nurses away from high-risk patients.
• Automation Bias: Staff felt pressured to comply with AI alerts over physical clinical evidence.
• Frontline Clinician Burnout: Staffing shortages and high workloads exacerbated the burden of false alerts.

Process and Methods

• Rigid Protocol Integration: AI alerts triggered rigid protocols that demanded compliance over judgment.
• Exclusion of Nurses in Design: Hospitals failed to consult frontline nursing staff during AI tool design.
• Lack of Real-world Validation: Hospitals deployed unvalidated tools that fell short of real-world promises.

Regulatory Environment

• No Centralized Gatekeeper: No single regulatory body validates AI tools before hospital deployment.
• Unregulated Algorithmic Tools: Hospitals bypassed traditional safeguards when introducing new AI technologies.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides a clear, detailed account of the specific clinical incident, the AI system involved, and the clinical reasoning used to avert harm. There is little ambiguity regarding the sequence of events or the role of the AI system.
• Ambiguities identified: The exact version of the Epic sepsis algorithm used at St. Rose Dominican Hospital at the time of the incident is not explicitly specified, though Epic's algorithm is discussed generally.
• Alternative interpretations: None. The incident is clearly a case of a clinical decision support system generating an inappropriate recommendation that was successfully intercepted by human oversight.

A clinical decision support AI at a US hospital generated an inappropriate sepsis treatment recommendation for a dialysis patient. The error was successfully intercepted by a nurse, preventing physical harm. The incident highlights localized clinical safety risks of AI integration in healthcare, with negligible implications for national security.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding AI integration in healthcare rather than an imminent national security crisis.
• Autonomy: Human-supervised. The AI acted as an assistant generating alerts, but final clinical decisions remained with human medical staff who successfully intervened.
• Novelty: Established threat. Errors and limitations in clinical decision support systems and sepsis-prediction algorithms are well-documented phenomena.

Impact by dimension

• Physical security: Negligible. Single clinical incident where potential patient harm was successfully averted by human intervention, resulting in no impact on critical infrastructure.
• Information security: Negligible. No involvement of intelligence capabilities, disinformation, or classified information compromise.
• Sovereignty: Negligible. No impact on state authority, territorial control, or core government decision-making processes.
• Economic security: Negligible. No threat to economic stability, strategic technology theft, or national technological competitiveness.
• Societal stability: Negligible. Localized clinical safety event with no implications for mass surveillance, human rights, or societal-scale stability.