Optum Algorithmic Health Risk Scores Reportedly Underestimated Black Patients' Needs

Researchers analyzing an Optum algorithm used by a large academic hospital reportedly found evidence that it tended to under-predict the health care needs of Black patients, which in turn appeared to result in lower prioritization for extra care programs compared with white patients exhibiting similar health profiles.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The algorithm's accuracy in predicting health needs was dependent on demographic group membership, performing significantly worse for Black patients by underestimating their severity of illness.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The discriminatory risk scores were generated by the deployed Optum algorithm, which unintentionally produced biased outputs due to using health-care costs as a proxy for medical needs.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used in healthcare as a diagnostic or triaging tool to allocate medical care, which has significant implications for safety and fundamental rights.

AI system and alleged parties

• AI system: Optum health risk algorithm (Optum)
• AI purpose: Resource Allocation; Medical Diagnosis Support
• Behaviour type: Assistant
• Alleged developer: Optum
• Alleged deployer: unnamed large academic hospital
• Alleged harmed parties: Black patients

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Minor
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Substantial
• Civil rights: direct Substantial, indirect Substantial
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Physical

Reported: The report does not explicitly describe direct physical injuries or deaths caused by the algorithm, but notes that excluded patients faced a greater chance of emergency room visits and hospitalizations.

Indirectly caused: The report notes that some residents in Arkansas experienced extreme disruptions and hospitalization due to sudden cuts from a different disabled-care algorithm, but does not explicitly document specific physical injuries directly caused by the Optum algorithm.

Inferred additional harm: It is highly likely that some of the millions of Black patients who were denied access to high-risk care management programs suffered worsened health outcomes, complications, or premature death, though these are not quantified in the reports.

Differential treatment

Reported: The report explicitly describes systematic differential treatment where Black patients were assigned lower risk scores than equally sick white patients.

Directly caused: The Optum algorithm systematically assigned lower risk scores to Black patients, reducing the proportion of Black patients referred for extra care by more than half (from an unbiased 46.5 percent to 17.7 percent).

Indirectly caused: The algorithm's bias resulted in healthier white patients being prioritized for intensive care management over sicker Black patients.

Inferred additional harm: This differential treatment likely occurred across the health systems managing care for up to 200 million people in the United States who utilized similar cost-based risk prediction tools.

Civil rights

Reported: The report explicitly describes the incident as a civil rights issue regarding equitable treatment in the health care system.

Directly caused: The algorithm systematically denied Black patients equal access to health care resources based on a biased metric, violating their right to non-discriminatory treatment.

Indirectly caused: The failure of regulatory bodies to oversee these clinical algorithms allowed systemic racial discrimination to be automated and scaled across major hospital systems.

Inferred additional harm: Widespread systemic violations of civil rights occurred across the US healthcare system for millions of Black patients whose care was guided by this or similar biased algorithms.

People affected

• Occurrences reported: 1
• People reportedly harmed: 50000
• People reportedly exposed: 3700000

Potential causes

Management

• Financial Incentive Prioritization: Systems focused on cost-containment incentives over equitable care delivery.
• Neglecting Tool Limitations: Management viewed algorithmic scores as substitutes for doctor expertise.
• Lack of Oversight Mechanisms: Organizations failed to establish mechanisms to detect algorithmic biases.

Technology

• Cost Proxy for Health Needs: Algorithm used healthcare costs to predict needs, ignoring clinical realities.
• Opaque Neural Networks: Image models learned to detect race through unknown visual shortcuts.
• Sepsis Prediction Failures: Sepsis detection tool failed to predict illness in 67 percent of patients.

Data Inputs

• Biased Historical Cost Data: Historical spending data reflected systemic disparities in healthcare access.
• Lack of Training Data Diversity: Approved AI tools lacked clear information on demographic training diversity.
• Unlabeled Racial Cues in Scans: Medical scans contained hidden racial patterns exploited by AI shortcuts.

Human Factors

• Lack of Designer Diversity: Lack of diversity among designers led to unmitigated algorithmic harms.
• Lack of Societal Context Training: Developers lacked training on social and historical contexts of health data.
• Overreliance on Off-The-Shelf Tools: Hospitals used tools without supplementing them with clinician expertise.

Process and Methods

• Proprietary Algorithm Opacity: Opaque software prevented researchers from auditing and detecting biases.
• Insufficient Pre-Deployment Audits: Developers failed to routinely audit algorithms for bias before deployment.
• Poor Clinician Procurement Input: Clinicians were not involved in purchasing decisions for clinical tools.

Regulatory Environment

• Under-regulated Clinical Software: FDA did not require review of administrative and risk-prediction tools.
• Non-mandatory Bias Testing: FDA only recommended rather than required testing for demographic bias.
• Lack of Public Performance Disclosures: Companies were not mandated to publicly release demographic performance data.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed, and peer-reviewed scientific evidence from a study published in Science regarding the Optum algorithm's bias, its mathematical cause, and its real-world impact on patient care.
• Ambiguities identified: There is some ambiguity regarding the exact number of patients directly affected at the specific hospital studied versus the broader national population of 70 million to 200 million lives.
• Alternative interpretations: Some industry analysts questioned whether the primary failure was the tool itself or how health systems used it as a sole decision-maker rather than one of many data elements.

A widely used healthcare risk prediction algorithm was found to systematically discriminate against Black patients by using cost as a proxy for medical need. While representing a major civil rights and critical infrastructure resource allocation concern affecting millions, the national security impact remains minor as it was a non-malicious system bias remediated through standard research and corporate intervention.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing, systemic algorithmic bias issue rather than an active, immediate national security crisis.
• Autonomy: Human-supervised. The AI acted as a decision-support tool, generating risk scores that influenced human clinical decisions, but did not execute treatment plans autonomously.
• Novelty: Evolved capability. Represents a significant, well-documented escalation of how algorithmic proxy metrics can automate and scale historical societal biases.

Impact by dimension

• Physical security: Minor. Algorithmic bias impacted resource allocation within the healthcare sector, a critical infrastructure, but did not involve kinetic attacks or active manipulation of control systems.
• Information security: Negligible. No evidence of information warfare, intelligence compromise, deepfakes, or classified data theft associated with this incident.
• Sovereignty: Negligible. The incident did not involve foreign AI systems operating in sovereign territory, compromise of core government decision-making, or disruption of constitutional processes.
• Economic security: Negligible. The algorithm affected healthcare spending metrics and resource distribution, but did not threaten overall economic stability, strategic industries, or technological competitive advantage.
• Societal stability: Minor. Systematic racial bias in healthcare allocation affected millions, representing a significant civil rights concern, but was managed through academic research and corporate remediation.