NarxCare’s Risk Score Model Allegedly Lacked Validation and Trained on Data with High Risk of Bias

NarxCare's overdose risk algorithm, lacking peer-reviewed validation, uses sensitive data like doctor visits, prescriptions, and possibly genetic information, leading to significant biases against women and Black patients. Factors like sexual abuse and criminal records exacerbate stigmas and disparities, often resulting in unjust denial of necessary pain medication. The newly approved AvertD genetic test shares similar issues, further complicating and potentially harming medical treatment decisions.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The primary failure is the AI system's inability to reliably and accurately assess addiction risk, resulting in high false-positive rates that harm patients.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Clinicians and pharmacists blindly rely on the algorithmic scores to deny medication, ignoring warnings that the tool should not replace medical judgment.
• 1.1 Unfair discrimination and misrepresentation: The algorithms incorporate biased data inputs, such as criminal records and confounded genetic markers, disproportionately impacting minority groups.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The harm is caused by the deployment of AI risk-scoring systems that unintentionally restrict access to healthcare for legitimate patients.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The systems are AI tools used in healthcare to assess patient risk, which directly impacts their access to essential medical treatment and pain management.

AI system and alleged parties

• AI system: AvertD, NarxCare (Bamboo Health, Solvd Health)
• AI purpose: Substance Abuse Prediction; Medical Diagnosis Support
• Behaviour type: Assistant
• Alleged developer: Appriss
• Alleged deployer: NarxCare, AvertD, Appriss
• Alleged harmed parties: American physicians, American pharmacists, American patients of minority groups, American patients

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Minor
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Substantial
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Substantial
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Physical

Reported: The report explicitly describes physical harm in the form of untreated severe pain and agony caused by the denial of necessary medications.

Indirectly caused: Patients like Amirault (hip replacement pain) and another patient with endometriosis were denied postsurgical and chronic pain medications, leaving them in severe physical agony.

Inferred additional harm: It is highly likely that thousands of chronic pain patients nationwide have suffered severe untreated physical pain, withdrawal, or turned to dangerous illicit street drugs due to being incorrectly flagged.

Financial

Reported: The report describes financial and professional ruin for a doctor whose practice was targeted based on prescribing data.

Indirectly caused: Dr. Pompy had his medical license suspended and faced federal prosecution (of which he was acquitted), resulting in severe financial loss and the closure of his practice.

Inferred additional harm: Other doctors and patients facing legal fees, loss of licenses, or lost workdays due to untreated pain or algorithmic flagging.

Differential treatment

Reported: The report explicitly describes differential treatment of medically complex patients and minority groups.

Directly caused: The algorithms generate higher risk scores for patients with complex medical needs (who see multiple doctors) and those of African American or Hispanic descent due to systemic biases in input data.

Indirectly caused: Black, Hispanic, and medically complex patients are disproportionately denied pain management and stigmatized by healthcare providers.

Inferred additional harm: Widespread systemic discrimination in healthcare access, reinforcing historical racial and socioeconomic inequalities.

Civil rights

Reported: The report explicitly describes violations of patients' rights to medical care and due process.

Indirectly caused: Patients were expelled from physician practices and denied postsurgical care with no way of appealing the decision or correcting errors in their NarxCare records.

Inferred additional harm: Thousands of patients losing bodily autonomy and access to essential healthcare without due process.

Privacy

Reported: The report explicitly describes privacy concerns regarding the aggregation of sensitive personal data.

Indirectly caused: Patients' private medical histories, prescription records, and veterinary records were aggregated and analyzed by a proprietary algorithm without clear consent.

Inferred additional harm: Millions of Americans having their private health data compiled into an opaque 'credit score' without their knowledge or consent.

Psychological

Reported: The report explicitly describes psychological distress, anxiety, and feelings of stigmatization and humiliation.

Indirectly caused: Patients reported feeling like 'suspects' under 'inquisition' and experienced severe anxiety and humiliation when interacting with medical staff who saw their high scores.

Inferred additional harm: Widespread anxiety, depression, and trauma among chronic pain patients who live in constant fear of losing access to their life-enabling medications.

People affected

• Occurrences reported: 1
• People reportedly harmed: 5
• People reportedly exposed: 150000000

Potential causes

Management

• Prioritizing Speed Over Accuracy: Platforms focus on rapid scoring rather than ensuring clinical safety.
• Contradictory Marketing Claims: Developers market wide data integration but claim only PDMP data is used.
• Lack of Developer Accountability: Companies deflect responsibility for clinical outcomes to individual doctors.

Technology

• Proprietary Black-Box Algorithms: Opaque models prevent external validation and verification of risk scoring.
• Low Algorithmic Precision: Low baseline rate of addiction leads to extremely high false-positive rates.
• Genetic Ancestry Confounding: AvertD genetic testing is confounded by population stratification, causing bias.

Data Inputs

• Confounded Veterinary Data: Animal prescriptions under owner names are flagged as human doctor-shopping.
• Complex Medical History Bias: Legitimate care for cancer or chronic illness mirrors doctor-shopping patterns.
• Biased Criminal Justice Data: Incorporating arrest records reinforces historical racial discrimination.

Human Factors

• Automation Bias by Clinicians: Medical staff blindly follow scores instead of exercising clinical judgment.
• Fear of Legal Prosecutions: Clinicians fear losing licenses or facing charges if they ignore high scores.
• Stigma Against Addiction: Prejudice leads to immediate patient dismissal without clinical verification.

Process and Methods

• Lack of Patient Redress: Patients have no clear pathway to correct erroneous data or appeal scores.
• No Independent Validation: Algorithms are widely deployed without peer-reviewed safety validation.
• Abrupt Care Termination: Clinics unilaterally dismiss patients without providing alternative pain care.

Regulatory Environment

• Absence of FDA Approval Mandate: NarxCare operates as a clinical tool without requiring FDA safety clearance.
• Contradictory FDA Approvals: FDA approved AvertD despite its advisory committee voting against it.
• Mandatory PDMP Query Laws: State laws force clinicians to consult unvalidated risk scoring platforms.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly detailed, from reputable investigative journalism sources, and cite specific academic studies, economic research, and multiple firsthand patient and physician accounts.
• Ambiguities identified: The exact proprietary formulas and weightings of NarxCare's algorithms remain secret, which is a noted issue but does not prevent classification.
• Alternative interpretations: None that are plausible given the consistent evidence of algorithmic failure across all reports.

The incident involves the domestic deployment of proprietary healthcare algorithms (NarxCare and AvertD) that generate biased opioid risk scores, leading to the denial of care. While causing individual distress and raising human rights concerns, the national security impact is minor and confined to domestic regulatory and societal policy domains.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic and regulatory concern rather than an immediate national security crisis.
• Autonomy: Human-supervised. The AI generates risk scores to assist clinical decisions, but healthcare professionals retain the ultimate authority to prescribe, albeit under regulatory pressure.
• Novelty: Evolved capability. Represents an advanced, widespread deployment of proprietary risk-scoring algorithms in critical public sectors, evolving from simpler automated systems.

Impact by dimension

• Physical security: Negligible. No kinetic threats, weapons deployment, or systemic physical compromise to critical infrastructure. Individual medical issues do not constitute a national security threat.
• Information security: Negligible. No indications of foreign intelligence compromise, deepfake operations, or systematic information warfare targeting national security institutions.
• Sovereignty: Negligible. Core government operations and sovereign territorial control remain unaffected. The issue is a domestic regulatory and healthcare policy challenge.
• Economic security: Negligible. No significant threat to strategic US industries, financial systems, or technological competitive advantage.
• Societal stability: Minor. Minor impact on societal stability. While the system exhibits bias and restricts access to medication for vulnerable populations, it remains a domestic policy issue manageable through standard legal and regulatory procedures.