UnitedHealth Accused of Deploying Allegedly Flawed AI to Deny Medical Coverage

UnitedHealthcare allegedly used a faulty AI algorithm with a 90% error rate to override doctors' recommendations and deny health coverage. This AI, developed by NaviHealth, reportedly led to premature discharge from care facilities and substantial out-of-pocket expenses for patients, according to a lawsuit filed in the District Court for Minnesota.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The nH Predict AI system failed to perform reliably or effectively by generating highly inaccurate, rigid predictions that resulted in a 90% error rate upon professional medical appeal.

Additional risk subdomains

• 5.2 Loss of human agency and autonomy: The AI's predictions overrode the clinical recommendations of treating physicians, and corporate targets stripped case managers of their decision-making discretion.
• 6.2 Increased inequality and decline in employment quality: Case managers faced disciplinary action or termination if they deviated from the AI's length-of-stay predictions to advocate for patients.

Causal factors

• Entity: AI
• Intent: Intentional
• Timing: Post-deployment

The financial and physical harms were caused by the systematic deployment of the nH Predict AI tool to automate coverage denials post-deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes an AI system used in healthcare to determine patient care and coverage, which falls under High Risk as it has significant implications for safety, fundamental rights, and public interests.

AI system and alleged parties

• AI system: nH Predict (UnitedHealth Group)
• AI purpose: Resource Allocation; Value Estimation
• Behaviour type: Assistant
• Alleged developer: NaviHealth
• Alleged deployer: UnitedHealthcare
• Alleged harmed parties: Medicare Advantage Plan Patients, Healthcare Providers (Doctors and Therapists), elderly patients

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Minor
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Substantial
• Civil rights: direct Negligible, indirect Substantial
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Physical

Reported: The report explicitly describes physical harm to at least one patient caused indirectly by the incident.

Indirectly caused: One elderly patient was discharged early due to coverage denial, fell at home after two days, and had to be taken by ambulance back to the nursing home.

Inferred additional harm: It is highly likely that among the tens of thousands of elderly patients prematurely discharged, many suffered physical setbacks, falls, or worsened medical conditions due to lack of proper rehabilitation, though specific numbers are not detailed.

Financial

Reported: The report explicitly describes substantial financial losses caused indirectly by the incident.

Indirectly caused: Patients and their families had to pay substantial out-of-pocket costs due to coverage denials: $150,000 for Lokken, $70,000 for Tetzloff, and $10,406.36 for the Connecticut patient, averaging $76,802.12 across these detailed cases.

Inferred additional harm: With tens of thousands of patients affected and only 0.2% appealing, the total financial loss to families depleting their savings is estimated to be in the hundreds of millions of dollars.

Differential treatment

Reported: The report explicitly describes differential treatment of individuals caused directly by the incident.

Directly caused: The AI system systematically targeted elderly Medicare Advantage patients, predicting shorter recovery times and denying coverage compared to what traditional Medicare would cover.

Indirectly caused: CMS warned that the use of such algorithms can reinforce and exacerbate existing demographic and socioeconomic biases in healthcare access.

Inferred additional harm: Elderly and vulnerable patients under Medicare Advantage plans received systematically worse coverage and care compared to those on traditional Medicare.

Civil rights

Reported: The report explicitly describes violations of human and civil rights caused indirectly by the incident.

Indirectly caused: The systematic denial of contractually and legally mandated healthcare coverage to elderly patients violates their basic rights to healthcare and fair treatment under government-funded programs.

Inferred additional harm: Widespread violations of patients' rights to individualized medical review and fair contract execution across the United States.

Psychological

Reported: The report explicitly describes psychological distress to patients and their families caused indirectly by the incident.

Indirectly caused: The lawsuit seeks damages for emotional distress, noting that families of seriously ill patients experienced significant anxiety and distress trying to appeal denials and facing the prospect of their elderly relatives being thrown out onto the street.

Inferred additional harm: Widespread anxiety and emotional trauma among thousands of elderly patients and their families facing sudden, unexpected coverage terminations during critical recovery periods.

People affected

• Occurrences reported: 1
• People reportedly harmed: 5
• People reportedly exposed: 15000000

Potential causes

Management

• Prioritizing Profits Over Care: Management used the tool to reduce care costs and increase corporate profits.
• Draconian Performance Targets: Managers forced employees to keep stays within 1 percent of AI predictions.
• Disciplinary Action for Deviation: Employees who deviated from the algorithm faced discipline or termination.

Technology

• Flawed nH Predict Algorithm: The model had a 90 percent error rate when predicting patient recovery times.
• Rigid Length-of-Stay Predictions: System spits out generic estimates that override treating doctors' judgments.
• Lack of Individual Nuance: The tool does not account for complex individual patient circumstances.

Data Inputs

• Limited Patient Data Fields: Model only uses a small set of patient factors like age and diagnosis.
• Historical Systemic Bias: The database of 6 million records inherits past cost-cutting inequalities.
• Exclusion of Comorbidities: Database ignores critical clinical events like catching COVID-19 or pneumonia.

Human Factors

• Lack of Patient Appeal Resources: Elderly patients lack the knowledge and resources to appeal denials.
• Overreliance on AI Projections: Medical reviewers and case managers treat algorithm predictions as gospel.

Process and Methods

• Automated Claims Denial Process: Insurers used automated tools to systematically cut off patient coverage.
• Proprietary Data Restrictions: Insurers refused to share nH Predict reports, claiming they are proprietary.
• Absence of Manual Reviews: Medical directors made adverse decisions without examining the patients.

Regulatory Environment

• Vague Medicare Advantage Rules: Vague rules allowed insurers to use predictive tools to deny care.
• Lack of CMS Enforcement: Absence of penalties and audits let insurers bypass original Medicare rules.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly detailed, citing specific legal filings, investigative journalism (STAT News), corporate statements, and regulatory responses (CMS memo).
• Ambiguities identified: The exact proprietary inner workings of the nH Predict algorithm are not fully public.
• Alternative interpretations: UnitedHealthcare claims the tool is only an advisory guide and that human medical directors make the final decisions, whereas the plaintiffs claim the AI effectively makes the decision due to strict corporate compliance targets.

UnitedHealthcare and NaviHealth faced legal and regulatory action for using the nH Predict AI tool to systematically deny post-acute care to Medicare Advantage patients. While causing physical and financial harm to individuals and disrupting federal program administration, the incident represents a regulatory and consumer protection issue with minor national security implications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: UnitedHealth Group

Threat characteristics

• Imminence: Long-term. This represents an ongoing regulatory and corporate governance concern regarding AI deployment in healthcare rather than an immediate national security crisis.
• Autonomy: Human-supervised. The AI system generates predictions that are meant to be reviewed by human case managers, though corporate pressure severely constrained human deviation from the AI recommendations.
• Novelty: Evolved capability. While automated insurance rules are established, using predictive algorithms to systematically override clinical physician decisions at scale represents an evolved and more sophisticated threat.

Impact by dimension

• Physical security: Minor. Healthcare is a critical infrastructure sector. The systematic use of an inaccurate AI tool to override clinical decisions for elderly patients introduces localized vulnerabilities in health delivery and patient safety, though it does not threaten national physical security.
• Information security: Negligible. No information warfare, intelligence compromise, or foreign disinformation campaigns are associated with this incident.
• Sovereignty: Minor. The incident impacted the administration of Medicare Advantage, a major federally funded program. The AI-driven denials undermined government healthcare objectives, prompting regulatory intervention by the Centers for Medicare and Medicaid Services.
• Economic security: Negligible. While causing significant individual financial harm, the incident does not threaten national economic stability, strategic industries, or technological competitive advantage.
• Societal stability: Minor. Systematic denial of contractually mandated healthcare to vulnerable elderly populations represents a localized civil rights and welfare concern, but does not threaten large-scale societal stability.