Researchers find evidence of racial, gender, and socioeconomic bias in chest X-ray classifiers

A study by the University of Toronto, the Vector Institute, and MIT showed the input databases that trained AI systems used to classify chest X-rays led the systems to show gender, socioeconomic, and racial biases.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The medical AI systems exhibit unequal diagnostic accuracy and performance across different demographic groups (gender, race, socioeconomic status) due to biased training datasets.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The diagnostic models lack robustness and fail to perform reliably when tested on diverse patient populations outside their primary training demographics.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The diagnostic AI systems unintentionally produce biased and incorrect classifications for certain demographic groups after being trained and deployed.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes AI used in healthcare, specifically 'diagnostic tools' like chest X-ray classifiers, which have significant implications for safety and fundamental rights.

AI system and alleged parties

• AI system: UnitedHealth Group algorithm, chest X-ray classifiers (UnitedHealth Group)
• AI purpose: Medical Diagnosis Support; Image Classification
• Behaviour type: Assistant
• Alleged developer: Qure.ai, Google, DarwinAI, Aidoc
• Alleged deployer: Mount Sinai Hospitals
• Alleged harmed parties: patients with Medicaid insurance, patients of minority groups, low-income patients, Hispanic patients, female patients

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Substantial
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment where AI models perform worse for female, minority, and low socioeconomic status patients.

Directly caused: The algorithms provided Medicaid patients with incorrect diagnoses at higher rates and underestimated by half the number of Black patients in need of greater care.

Indirectly caused: Underrepresented racial groups are subject to less reliable eye-disease and skin-cancer diagnoses due to datasets being dominated by North American, European, and light-skinned patient data.

Inferred additional harm: Widespread deployment of these systems likely perpetuates and exacerbates systemic healthcare inequalities for millions of marginalized patients globally.

People affected

• Occurrences reported: 1
• People reportedly exposed: 129000

Potential causes

Management

• Missing Dataset Disclaimers: Failure to provide clear disclaimers regarding dataset collection limitations.

Technology

• Manufacturer Format Bias: Algorithms became biased to scan formats of specific machine manufacturers.
• Erroneous NLP Labeling: Labels extracted using NLP from radiology reports contained errors.

Data Inputs

• Imbalanced Training Datasets: Datasets overrepresented white, male, and wealthy patient demographics.
• Geographic Data Bias: Most data originated from limited regions, reducing global performance.
• Device Quality Variations: Variations in imaging device quality across sites confounded diagnostic models.

Human Factors

• Differential Healthcare Access: Minorities use healthcare less, reducing their representation in datasets.

Process and Methods

• Insufficient Peer Review: Classifiers lacked thorough peer review before real-world deployment.
• Lack of Fairness Analysis: Practitioners did not run rigorous fairness analyses before deployment.
• Reticence to Share Code: Withholding code and datasets prevented external verification and auditing.

Information quality

• Classification confidence: High
• Reason for confidence: The report is based on peer-reviewed scientific studies from reputable institutions (MIT, Toronto, Stanford) detailing specific dataset sizes and bias metrics. The findings of demographic disparities are clearly documented with statistical context.
• Ambiguities identified: The report does not specify the exact number of real-world patients who received incorrect treatment due to these biases, focusing instead on dataset and model evaluation.
• Alternative interpretations: None. The evidence of algorithmic bias across demographic groups is robust and consistent across multiple cited studies.

A scientific study revealed systematic racial, gender, and socioeconomic biases in medical diagnostic AI systems trained on public datasets. While highlighting significant concerns regarding healthcare equity and algorithmic discrimination, the national security implications are minor, with risks manageable through standard regulatory, medical, and scientific validation frameworks.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: North America, Europe, China
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing, systemic challenge in AI development and training data curation rather than an active, time-sensitive crisis.
• Autonomy: Human-supervised. Medical diagnostic AI models act as decision-support systems where clinicians ultimately make and execute final medical decisions.
• Novelty: Established threat. Algorithmic bias and demographic disparities in machine learning models are well-documented and heavily researched phenomena.

Impact by dimension

• Physical security: Negligible. The incident involves medical diagnostic software biases and does not present any kinetic, physical, or critical infrastructure security threats.
• Information security: Negligible. No evidence of information warfare, intelligence compromise, espionage, or systematic disinformation campaigns.
• Sovereignty: Negligible. The incident does not impact state authority, border controls, elections, or core government decision-making processes.
• Economic security: Negligible. While involving major tech firms, the incident does not pose a threat to financial systems, strategic supply chains, or national technological advantages.
• Societal stability: Minor. Systematic performance disparities and biases against marginalized groups in medical AI represent unequal treatment, but the impact is manageable within standard regulatory and medical validation procedures.