Length of Stay False Diagnosis Cut Off Insurer's Payment for Treatment of Elderly Woman

An elderly Wisconsin woman was algorithmically determined to have a rapid recovery, an output which the insurer based on to cut off payment for her treatment despite medical notes showing her still experiencing debilitating pain.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The nH Predict algorithm fails to perform reliably or effectively under varying patient conditions, generating rigid, speculative predictions that do not account for individual clinical realities.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Insurers overrely on the predictive algorithm as a hard-and-fast rule to deny care, ignoring individual clinical realities.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the nH Predict AI system's inaccurate and rigid predictions of patient recovery times after its deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used in healthcare and insurance decision-making to determine patient care and coverage, which has significant implications for the safety and fundamental rights of vulnerable elderly patients.

AI system and alleged parties

• AI system: nH Predict
• AI purpose: Resource Allocation; Health AI Assistant
• Behaviour type: Assistant
• Alleged developer: NaviHealth
• Alleged deployer: Security Health Plan, NaviHealth
• Alleged harmed parties: Frances Walter, elderly patients

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Minor
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Substantial
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Physical

Reported: The report explicitly describes physical harm to patients whose rehabilitation care was prematurely terminated.

Indirectly caused: Patients like [PERSON_001] and Dolores Millam had their rehabilitation care cut off while still unable to walk or care for themselves, delaying their recovery and leaving them at safety risks.

Inferred additional harm: Thousands of elderly patients likely suffered prolonged illness, lack of recovery, or physical regression due to premature discharge and denied rehabilitation therapies.

Financial

Reported: The report describes significant financial losses, including one patient spending down her life savings and another receiving a bill of almost $40,000.

Indirectly caused: Dolores Millam's family incurred a $40,000 nursing home bill due to the denial, and [PERSON_001] lost thousands of dollars and her life savings before the denials were overturned.

Inferred additional harm: Millions of dollars in collective financial losses for thousands of families who had to pay out-of-pocket for nursing home care or spend down savings to qualify for Medicaid.

Differential treatment

Reported: The report explicitly describes systematic differential treatment of elderly Medicare Advantage patients compared to traditional Medicare patients.

Directly caused: The nH Predict algorithm targeted elderly patients in Medicare Advantage plans, generating rigid discharge dates that resulted in systematic denials of care.

Indirectly caused: Medicare Advantage beneficiaries experienced unequal access to rehabilitation and nursing care compared to those on traditional Medicare.

Inferred additional harm: Systematic unequal treatment of millions of elderly and disabled Medicare Advantage beneficiaries who are subjected to automated, proprietary coverage restrictions.

Psychological

Reported: The report explicitly describes patients and families experiencing severe distress, crying, and the nightmare of fighting denials.

Indirectly caused: Patients like the stroke victim were left 'drooling and crying' in distress, and families like Dolores Millam's daughter made hundreds of phone calls in panic trying to resolve the sudden cutoff of care.

Inferred additional harm: Widespread anxiety, stress, and emotional trauma among thousands of elderly patients and their families facing sudden, unexpected medical bills and loss of care.

People affected

• Occurrences reported: 1
• People reportedly harmed: 150000
• People reportedly exposed: 31000000

Potential causes

Management

• Profit Prioritization over Care: Insurers acquire AI tools to aggressively cut costs and boost profits.
• Lack of Quality Controls: Management failed to implement quality controls over AI-driven determinations.

Technology

• Proprietary Black-Box Algorithm: nH Predict algorithm is proprietary, preventing independent review of its logic.
• Untested Real-World Performance: NaviHealth published no scientific studies assessing algorithm performance.

Data Inputs

• Averaged Statistical Database: Database of 6 million patients abstracts individuals into generic statistics.
• Ignoring Individual Comorbidities: Input data fails to account for complex, individual patient health histories.

Human Factors

• Blind Reliance on Predictions: Insurers treat suggestive predictions as hard-and-fast rules for discharge.
• Inability to Challenge Algorithm: Patients and doctors are unable to see or question the AI calculations.

Process and Methods

• Lack of Algorithmic Transparency: AI reports are run secretly and not shared with patients or providers.
• Rigid Prior Authorization Process: Insurers use AI to systematically deny care and force long appeals.

Regulatory Environment

• Unregulated Insurance Algorithms: AI tools used for coverage decisions lack FDA or government oversight.
• Permissive Medicare Rules: Current regulations allow insurers to use internal proprietary criteria.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides extensive, high-quality investigative journalism from STAT, including specific patient case studies, court rulings, and statistics on appeals, which clearly document the role of the nH Predict algorithm in driving insurance denials.
• Ambiguities identified: The exact proprietary mechanics of the nH Predict algorithm and the precise degree of automated vs. human decision-making in the denial process remain somewhat opaque due to corporate secrecy.
• Alternative interpretations: Insurers argue the tool is merely a non-binding guide and that denials are based on standard Medicare criteria, suggesting the harm is entirely due to human administrative decisions rather than the AI itself.

Private insurers utilized a proprietary AI algorithm, nH Predict, to automate and restrict post-acute care coverage for elderly Medicare Advantage patients. While the incident has caused significant financial and physical distress for thousands of patients and prompted federal regulatory intervention, its direct implications for national security remain minor and manageable within standard domestic regulatory frameworks.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This is an ongoing systemic issue regarding commercial AI deployment in healthcare administration, rather than an active national security crisis.
• Autonomy: Human-supervised. The AI generates predictive assessments which are then reviewed and implemented by insurance coordinators and medical directors, though often rubber-stamped.
• Novelty: Evolved capability. Represents a significant advancement in the scale and integration of proprietary predictive modeling within private insurance decision-making systems.

Impact by dimension

• Physical security: Negligible. The incident involves an insurance coverage algorithm and does not pose a direct threat to physical systems, critical infrastructure, or kinetic safety.
• Information security: Negligible. There is no indication of intelligence compromise, classified data theft, or information warfare operations associated with this commercial healthcare algorithm.
• Sovereignty: Minor. The algorithm impacts the administration of Medicare Advantage, a government-funded program, prompting regulatory responses from federal Medicare officials to limit its use.
• Economic security: Negligible. While causing individual financial harm to patients, the incident does not threaten national economic stability, strategic industries, or technological competitive advantage.
• Societal stability: Minor. The system results in systematic differential treatment and financial distress for elderly patients, but does not threaten overall societal stability or constitute state-level algorithmic oppression.