Frontline workers protest at Stanford after hospital distributed vaccine to administrators

In 2020, Stanford Medical Center's distribution algorithm only designated 7 of 5,000 vaccines to Medical Residents, who are frontline workers regularly exposed to COVID-19.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The algorithm failed to perform effectively under the condition of residents rotating departments, leading to a flawed prioritization list.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The algorithm's age-based scoring system systematically disadvantaged younger frontline residents relative to older, non-frontline staff.
• 5.1 Overreliance and unsafe use: Hospital leadership initially relied on the algorithmic output and failed to correct the distribution list even after being alerted to the errors.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the Stanford vaccine distribution algorithm post-deployment, leading to an unintentional and unexpected deprioritization of frontline residents.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is an algorithmic tool used in healthcare to allocate critical medical resources (vaccines), which has significant implications for the safety and health of essential workers.

AI system and alleged parties

• AI system: Stanford vaccine algorithm
• AI purpose: Resource Allocation; Workforce Administration
• Behaviour type: Tool
• Alleged developer: Stanford Medical Center
• Alleged deployer: Stanford Medical Center
• Alleged harmed parties: Stanford Medical residents, Stanford Medical frontline workers

Harm severity

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes systematic differential treatment of frontline residents compared to remote-working staff.

Directly caused: Only seven out of over 1,300 frontline residents were prioritized for the first 5,000 vaccine doses, while administrators and remote-working doctors were selected.

Indirectly caused: N/A

Inferred additional harm: N/A

Psychological

Reported: The reports describe residents feeling shocked, angry, and feeling that the hospital did not care about them.

Directly caused: Residents experienced significant anger, shock, and distress upon learning they were excluded from the first wave of vaccinations despite working on the front lines.

Indirectly caused: N/A

Inferred additional harm: Widespread moral injury and anxiety among the 1,300 residents and fellows who felt abandoned by their institution during a pandemic surge.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1300
• People reportedly exposed: 1300

Potential causes

Management

• Dismissal of Early Warnings: Leadership ignored flaws flagged on Tuesday and proceeded with rollout.
• Prioritizing Automation Over Logic: Relied on algorithm outputs instead of manual verification by chiefs.

Technology

• Flawed Rules-Based Formula: Age-based scoring and complex weights disadvantaged middle-aged residents.
• Overcomplicated Algorithm: Too many variables made the system hard to understand and verify.

Data Inputs

• Proxy Variables For Risk: Used department testing rates instead of actual direct patient exposure.
• Rotation Data Exclusion: Rotating residents lacked a single department assignment, losing points.

Human Factors

• Lack of Stakeholder Input: Residents and department chairs were excluded from the design process.
• Overreliance on Proxies: Designers assumed department testing rates reflected individual exposure risk.

Process and Methods

• Lack of Outcome Verification: System output was not validated against real frontline status before rollout.
• No System Override Mechanism: Hospital lacked a clear mechanism to quickly adjust or fix flawed outputs.

Regulatory Environment

• Misapplied CDPH Guidelines: State guidelines were meant for counties, not for internal hospital triage.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the algorithm's variables, the specific number of residents affected, and the organizational response. There is high consensus across multiple reputable news sources regarding the facts of the incident.
• Ambiguities identified: The exact mathematical weights of the algorithm's variables are not fully detailed, and the precise definition of some job-based variables remains unclear.
• Alternative interpretations: The incident could be viewed purely as a human management/ethical failure in designing the rules, rather than an AI failure, given that it was a simple rules-based formula rather than machine learning.

In December 2020, Stanford Medicine's rules-based vaccine allocation algorithm failed to prioritize frontline medical residents, leading to localized protests and a subsequent policy revision. The incident was an internal, unintentional administrative error with negligible implications for national security.

• Overall national security impact: Negligible
• Response level: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident was a localized, quickly resolved administrative error with no ongoing national security threat or active crisis.
• Autonomy: Human-controlled. The rules-based algorithm acted as a deterministic tool to assist human decision-makers, who retained final authority and ultimately overrode the system.
• Novelty: Established threat. Flaws in data inputs and rules-based formulas leading to administrative errors are well-established issues rather than novel technological threats.

Impact by dimension

• Physical security: Negligible. The incident was a localized vaccine distribution error at a single hospital. It did not cause systemic disruption to national critical infrastructure or healthcare systems.
• Information security: Negligible. No information warfare, intelligence compromise, or adversarial disinformation campaigns were associated with this internal administrative incident.
• Sovereignty: Negligible. The incident involved a private medical center's internal resource allocation and did not impact state sovereignty or core government functions.
• Economic security: Negligible. The algorithmic failure did not target financial systems, result in strategic technology theft, or impact national economic security.
• Societal stability: Negligible. While the incident caused localized protests and distress among hospital staff, it posed no threat to national societal stability or civil liberties.