Northpointe Risk Models

An algorithm developed by Northpointe and used in the penal system is two times more likely to incorrectly label a black person as a high-risk re-offender and is two times more likely to incorrectly label a white person as low-risk for reoffense according to a ProPublica review.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The COMPAS algorithm disproportionately misclassified black defendants as high-risk and white defendants as low-risk, leading to systemic discrimination and unequal outcomes.

Additional risk subdomains

• 7.4 Lack of transparency or interpretability: The proprietary nature of the COMPAS algorithm prevented public audit and made it impossible for defendants to challenge the basis of their scores.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The biased risk scores were generated by the COMPAS AI system post-deployment, representing an unintentional outcome of using historical arrest data.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used in law enforcement and administration of justice, specifically for risk assessments and sentencing, which has significant implications for fundamental rights and public interests.

AI system and alleged parties

• AI system: COMPAS (Northpointe)
• AI purpose: Recidivism Prediction; Judicial Decision Support
• Behaviour type: Assistant
• Alleged developer: Northpointe
• Alleged deployer: Northpointe
• Alleged harmed parties: Accused People

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Substantial
• Civil rights: direct Substantial, indirect Substantial
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes

Indirectly caused: Defendants labeled high-risk faced higher bail amounts or detention, leading to lost wages and legal costs, though specific dollar amounts are not quantified in the reports.

Inferred additional harm: Unjustified pretrial detention likely caused millions of dollars in lost wages, job losses, and legal fees for thousands of affected defendants.

Differential treatment

Reported: Yes

Directly caused: The COMPAS algorithm directly assigned higher risk scores to black defendants compared to similarly situated white defendants, with black non-recidivists being twice as likely to be rated high-risk (45% vs 23%).

Indirectly caused: Judges using these biased scores as recommendations made harsher bail and sentencing decisions for black defendants.

Inferred additional harm: Systemic racial disparities in pretrial release and sentencing were reinforced and automated across multiple jurisdictions in the United States.

Civil rights

Reported: Yes

Directly caused: The biased risk scores violated defendants' rights to equal protection and due process, as highlighted in the Loomis v. Wisconsin case where the proprietary nature of the tool prevented a proper challenge.

Indirectly caused: Unjustified deprivation of liberty through biased pretrial detention and inflated prison sentences.

Inferred additional harm: Widespread civil rights violations occurred as thousands of individuals were detained or sentenced based on opaque, racially biased algorithmic recommendations.

Psychological

Reported: Yes

Indirectly caused: The stress and anxiety of being wrongfully classified as high-risk and potentially facing longer pretrial detention or harsher sentencing.

Inferred additional harm: It is likely that thousands of defendants experienced significant psychological distress, anxiety, and trauma due to unjustified detention or harsher sentencing resulting from biased scores.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1154
• People reportedly exposed: 18610

Potential causes

Management

• Prioritizing Proprietary Secrets: Vendor prioritized intellectual property over public accountability.
• Inadequate Risk Assessments: Deploying tools before rigorously evaluating societal and racial impacts.
• Neglecting Demographic Specifics: Failing to implement custom models for specific groups like women.

Technology

• Opacity of Proprietary Algorithms: COMPAS is a trade secret, preventing public audit or defense scrutiny.
• Inherent Mathematical Trade-offs: Impossible to satisfy all fairness definitions when base rates differ.
• Low Predictive Accuracy: Recidivism predictions are only slightly better than a coin flip.

Data Inputs

• Biased Historical Arrest Data: Arrest data reflects systemic racial disparities and over-policing.
• Proxy Variables for Protected Classes: Questions on family and neighborhood act as proxies for race and class.
• Inaccurate Data Entry: Name and birthdate errors led to incorrect criminal record matching.

Human Factors

• Automation Bias by Judges: Judges may over-rely on risk scores without understanding limitations.
• Lack of Algorithmic Literacy: Users and defendants cannot interpret or challenge complex risk scores.
• Societal Stereotypes in Research: Researchers utilized facial features to infer criminality without ethics.

Process and Methods

• Arrests Used as Proxy for Crime: Using arrest records instead of actual convictions distorts risk scores.
• Lack of Independent Validation: Many states use these tools without conducting validation processes.
• Flawed Research Methodology: Criminality studies lacked criminology and physiology expert input.

Regulatory Environment

• Absence of Algorithmic Audits: Lack of federal mandates to audit machine learning tools for bias.
• Weak Transparency Standards: No legal requirement to disclose source code of public decision tools.
• Unresolved Legal Definitions: Courts struggle to define constitutional fairness in mathematical terms.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide extensive, detailed statistical data from ProPublica's investigation, academic studies, and legal cases (Loomis v. Wisconsin). The core mechanism of the bias and its real-world impact are well-documented and consistent across sources.
• Ambiguities identified: There is some academic debate on the mathematical definition of fairness and whether the bias is a mechanical result of differing base rates of recidivism rather than algorithmic malice.
• Alternative interpretations: Some researchers argue the tool is statistically unbiased because it achieves predictive parity (scores mean the same probability of reoffending regardless of race), and that the disparate error rates are mathematically inevitable.

The COMPAS incident highlighted systemic racial bias in automated risk assessment tools used within the U.S. judicial system. While representing a significant civil rights and societal fairness concern, its direct national security impact remains minor, as it does not threaten critical infrastructure, sovereignty, or national defense and is managed through domestic legal frameworks.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The issue represents an ongoing, systemic challenge regarding algorithmic fairness in public policy rather than an active, imminent crisis.
• Autonomy: Human-supervised. The COMPAS tool acts as a decision-support assistant, providing risk scores to human judges who make the final legal determinations.
• Novelty: Evolved capability. The incident represents a significant escalation in the deployment and public exposure of biased automated decision-making in the public sector.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic capabilities were targeted or affected by the recidivism risk assessment tool.
• Information security: Negligible. The incident did not involve information warfare, deepfake operations targeting leadership, or compromise of intelligence capabilities.
• Sovereignty: Minor. The tool influenced state-level judicial decisions and court processes, representing a minor impact on government functions managed through standard legal appeals.
• Economic security: Negligible. There was no threat to strategic industries, semiconductor supply chains, or national technological competitive advantage.
• Societal stability: Minor. The algorithm introduced systematic racial bias into sentencing decisions, affecting civil liberties, but did not threaten national security or cause civil unrest.