Algorithmic Allocation of Resources in Healthcare for Disabled and Elderly Care Services Allegedly Harming Patients

A healthcare algorithm designed to equitably distribute caregiving resources drastically cut care hours for the disabled and elderly, leading to significant hardships and harm. Initially developed for fair resource allocation, the system ultimately faced legal challenges for its inability to accurately assess individual needs, resulting in reduced essential care and raising ethical concerns about AI in healthcare decision-making.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The algorithm failed to perform reliably or effectively under varying conditions, such as failing to factor in specific medical diagnoses like cerebral palsy, leading to wildly irrational and harmful care allocations.

Additional risk subdomains

• 5.2 Loss of human agency and autonomy: The algorithm replaced the professional judgments of nurses and social workers, and assessors blindly deferred to the computer's outputs without meaningful recourse for patients.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The severe budget cuts and subsequent physical neglect were caused by the decisions of the deployed algorithmic resource allocation system, which produced unexpected harmful outcomes while pursuing the goal of resource distribution.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used by public authorities to evaluate eligibility and allocate public assistance benefits and services, which has significant implications for safety and fundamental rights.

AI system and alleged parties

• AI system: algorithmic home care assessment system
• AI purpose: Resource Allocation
• Behaviour type: Tool
• Alleged developer: State governments, Brant Fries
• Alleged deployer: Washington DC government, State governments, Pennsylvania state government, Missouri state government, Iowa state government, Idaho state government, Arkansas state government
• Alleged harmed parties: Tammy Dobbs, Low-income people, Larkin Seiler, Elderly people, Disabled people

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Substantial
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Negligible
• Civil rights: direct Substantial, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Physical

Reported: The report explicitly describes severe physical harm and neglect, including cases of death, resulting from the care cuts.

Indirectly caused: The reduction in care hours caused patients to go without bathing, miss treatments, face increased risk of falling, lie in their own waste, develop bed sores, and skip meals. In Washington DC, some clients reportedly died due to service cuts.

Inferred additional harm: Given that thousands of people across more than a dozen states had their care budgets cut, it is highly likely that hundreds of other vulnerable individuals suffered undocumented physical neglect, injuries, or worsened medical conditions.

Financial

Reported: The report describes individual home care budgets being cut in half, though it does not provide specific total dollar amounts.

Indirectly caused: Vulnerable individuals had their state-funded care budgets drastically reduced, forcing them to either go without care or seek alternative, potentially costly support.

Inferred additional harm: Significant financial strain was likely placed on thousands of families who had to privately fund care or reduce working hours to care for affected relatives.

Differential treatment

Reported: The report describes arbitrary and unequal allocation of care hours among disabled individuals.

Directly caused: The algorithm sorted patients into arbitrary levels of need, failing to factor in specific conditions like cerebral palsy or diabetes, resulting in unequal and unfair treatment.

Inferred additional harm: Thousands of disabled individuals with complex needs were treated inequitably compared to those whose needs happened to align better with the simplistic scoring system.

Civil rights

Reported: The report describes violations of constitutional rights, as determined by court rulings in Idaho and Arkansas.

Directly caused: The arbitrary cuts deprived individuals of their basic freedom, independence, and right to receive necessary medical care in their homes.

Inferred additional harm: Widespread violations of due process and the right to live independently likely occurred across the dozen states implementing these unscrutinized algorithms.

Psychological

Reported: The report explicitly describes severe distress, fear, and mental suffering among the affected individuals.

Indirectly caused: Affected individuals experienced immense fear of being institutionalized, loss of independence, and severe distress, with one patient describing going 'berserk' upon hearing the decision.

Inferred additional harm: Widespread anxiety, depression, and feelings of helplessness were likely experienced by thousands of elderly and disabled patients facing sudden, drastic reductions in their daily support systems.

People affected

• Occurrences reported: 1
• People reportedly harmed: 2000
• People reportedly exposed: 10000

Potential causes

Management

• Prioritizing Cost Over Safety: States deployed crude algorithms to ration scarce care budgets arbitrarily.

Technology

• Flawed Algorithmic Calculations: The algorithm failed to factor in critical conditions like cerebral palsy.
• Binary Scoring Sensitivity: A single point score difference drastically changed the allocated care hours.

Data Inputs

• Incorrect Assessor Inputs: Assessors made errors, like marking amputees as having no mobility issues.
• Inflexible Questionnaire Data: The 286-question assessment failed to capture individual health subtleties.

Human Factors

• Overreliance on Automation: Assessors blindly followed computer outputs without applying clinical judgment.

Process and Methods

• Black Box Decision Making: Care decisions lacked transparency, making them extremely difficult to contest.
• Inadequate Algorithmic Calibration: The system was designed to divide resources rather than assess actual needs.

Regulatory Environment

• Lack of Algorithmic Oversight: No regulatory standards or oversight existed to govern healthcare algorithms.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, detailed accounts of the algorithm's deployment, its specific failures, the states involved, and the resulting physical and legal outcomes. The roles of the AI system and human actors are explicitly defined.
• Ambiguities identified: The exact mathematical formula of the algorithm is not detailed, and the precise number of total affected individuals across all states is described generally as 'thousands' rather than a specific count.
• Alternative interpretations: The incident could be viewed primarily as a policy and funding failure rather than an AI failure, though the algorithm's specific technical flaws and its role as the sole decision-maker support the AI safety classification.

An algorithmic resource allocation system used by several US states caused severe, arbitrary cuts to home care for disabled and elderly citizens, resulting in physical neglect and legal challenges. From a national security perspective, the impact is minor, representing a domestic administrative and civil rights issue that was successfully addressed through the US judicial system.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident spans a historical period (2008-2018) and has been largely addressed through judicial interventions, representing no immediate national security crisis.
• Autonomy: Human-supervised. State assessors entered data and state officials executed the decisions, though they exhibited high automation bias by blindly deferring to the algorithm.
• Novelty: Established threat. Flawed algorithmic decision-making and automated resource allocation systems in public administration represent a known and documented class of software failure.

Impact by dimension

• Physical security: Negligible. The incident involved domestic healthcare resource allocation and did not impact critical infrastructure or kinetic national defense systems.
• Information security: Negligible. No intelligence compromise, disinformation campaigns, or espionage activities were associated with this domestic administrative tool.
• Sovereignty: Minor. The system disrupted state-level healthcare administration and led to unconstitutional actions, but impacts were resolved through domestic legal and judicial processes.
• Economic security: Negligible. The incident affected individual healthcare budgets and state funding allocation, but posed no threat to national economic stability or strategic technology advantages.
• Societal stability: Minor. The system caused severe distress and physical neglect among vulnerable populations across several states, representing a domestic human rights concern managed via civil litigation.