Inefficiencies in the United States Resident Matching Program

Alvin Roth, a Ph.D at the University of Pittsburgh, describes the National Resident Matching Program (NRMP) and suggests future changes that are needed in the algorithm used to match recently graduated medical students to their residency programs.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The GPT-3 model demonstrated a critical lack of capability, consistency, and medical expertise during healthcare testing, leading to highly unsafe recommendations.

Additional risk subdomains

• 1.2 Exposure to toxic content: The chatbot generated extremely harmful and unsafe content by suggesting that a mock patient should commit suicide.
• 2.2 AI system security vulnerabilities and attacks: The reports also detail a Tesla Autopilot lane recognition vulnerability to adversarial attacks.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Pre-deployment

The risk was caused by the GPT-3 model's output during an experimental pre-deployment evaluation by Nabla, where it unintentionally generated a harmful suggestion.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system was evaluated for 'diagnosis support, and treatment recommendation' in healthcare, which is a high-risk application under the EU AI Act due to its potential impact on patient safety.

AI system and alleged parties

• AI system: GPT-3
• AI purpose: Medical Diagnosis Support; Chatbot
• Behaviour type: Assistant
• Alleged developer: National Resident Matching Program
• Alleged deployer: National Resident Matching Program
• Alleged harmed parties: Medical Residents

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1

Potential causes

Technology

• Inconsistency in LLM Outputs: GPT-3 showed high inconsistency and lacked scientific and medical expertise.
• Adversarial Input Vulnerability: Tesla Autopilot lane recognition was vulnerable to three small stickers.
• Weakness in Semantic Reasoning: AI systems in the Winograd Challenge succeeded only 3% above random chance.

Data Inputs

• Adversarial Physical Samples: Small stickers placed in intersections manipulated the lane recognition system.
• Lack of Domain-Specific Data: GPT-3 lacked specialized medical training data needed for clinical tasks.
• Manipulated Prompt Inputs: Human editors manipulated prompt inputs to generate sensationalist threats.

Human Factors

• Active Human Manipulation: Human editors manipulated GPT-3 to exaggerate harm and generate threats.
• Risky Experimental Testing: Mock patients or researchers tested the bot with dangerous scenarios.
• Student Ethics Oversight Failure: A student deployed a facial recognition app without addressing privacy.

Process and Methods

• Lack of Deployment Guardrails: Medical chatbot was tested without safety filters preventing harmful advice.
• Inadequate Vulnerability Testing: Autopilot systems lacked robust testing against physical adversarial attacks.
• Uncontrolled Campus Deployment: App was deployed on-campus before resolving ethical and privacy concerns.

Information quality

• Classification confidence: High
• Reason for confidence: The reports clearly state that the incidents described were downgraded because they represent experimental findings, academic studies, or projected risks rather than real-world harm events. This explicit context allows for a highly confident assessment that no actual harm occurred to real people.
• Ambiguities identified: None. The reports are clear about the experimental nature of the GPT-3 medical chatbot and the reasons for downgrading the other incidents.
• Alternative interpretations: One could interpret the reports as describing potential future harms, but the text explicitly confirms these are not real-world harm events.

A collection of downgraded AI safety cases, including pre-deployment testing of a GPT-3 medical chatbot and academic research on computer vision vulnerabilities. Because these occurred in experimental, academic, or controlled settings with no real-world deployment or malicious intent, they carry negligible national security implications.

• Overall national security impact: Negligible
• Scope: Multiple nations
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. These represent academic, experimental, and mitigated cases, reflecting long-term strategic safety concerns rather than active crises.
• Autonomy: Human-supervised. The systems were tested in controlled, human-supervised environments or heavily edited by human operators before any public exposure.
• Novelty: Established threat. LLM output instability, adversarial attacks on computer vision, and facial recognition privacy risks are well-documented and established research areas.

Impact by dimension

• Physical security: Negligible. The incidents represent pre-deployment experiments and academic vulnerabilities with no actual physical harm or critical infrastructure damage reported.
• Information security: Negligible. The GPT-3 generated Guardian op-ed was heavily edited by humans, and other incidents involve student projects or private tests with no intelligence compromise.
• Sovereignty: Negligible. No core government operations, electoral systems, or sovereign decision-making processes were impacted by these academic and experimental cases.
• Economic security: Negligible. The findings represent standard risk identification during pre-deployment testing and academic research, with no loss of strategic technology or economic stability.
• Societal stability: Negligible. Although the medical chatbot generated highly unsafe suggestions, it occurred in a closed, experimental setting on a mock patient, causing no real-world societal harm.