COMPAS Algorithm Reportedly Performs Poorly in Crime Recidivism Prediction

Correctional Offender Management Profiling for Alternative Sanctions (COMPAS), a recidivism risk-assessment algorithmic tool used in the judicial system to assess likelihood of defendants' recidivism, is found to be less accurate than random untrained human evaluators.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The COMPAS algorithm systematically treats black defendants unequally by falsely flagging them as high-risk at twice the rate of white defendants, leading to unfair sentencing and parole outcomes.

Additional risk subdomains

• 7.4 Lack of transparency or interpretability: The proprietary nature of COMPAS and TrueAllele prevents defendants and independent auditors from inspecting the source code, violating due process.
• 7.3 Lack of capability or robustness: COMPAS was demonstrated to perform no better than untrained human volunteers or a simple two-variable linear predictor, showing a lack of capability.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk of racial bias and inaccurate predictions stems directly from the outputs of the deployed AI systems, which was an unexpected and unintentional outcome of their design.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The EU AI Act classifies AI systems used in law enforcement, migration, asylum, and border control management, as well as administration of justice and democratic processes (such as risk assessment tools used by judicial authorities), as High Risk (Level 2).

AI system and alleged parties

• AI system: COMPAS (Equivant)
• AI purpose: Recidivism Prediction; Judicial Decision Support
• Behaviour type: Assistant
• Alleged developer: Equivant
• Alleged deployer: Equivant
• Alleged harmed parties: Accused People

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Substantial
• Civil rights: direct Substantial, indirect Substantial
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: Yes

Directly caused: The COMPAS algorithm directly caused differential treatment by falsely flagging black defendants as high risk at almost twice the rate of white defendants (40% vs 25% false positive rate), leading to harsher bail and sentencing recommendations.

Indirectly caused: Systemic biases in arrest data were reinforced and amplified by the AI, leading to indirect systemic discrimination against black defendants in the judicial system.

Inferred additional harm: It is likely that thousands of black defendants across the US experienced harsher judicial outcomes due to these biased algorithmic assessments.

Civil rights

Reported: Yes

Directly caused: The use of proprietary, opaque algorithms like COMPAS and TrueAllele directly violated defendants' due process rights by preventing them from examining or challenging the evidence and scoring methodologies used against them in court.

Indirectly caused: The lack of transparency and inability to cross-examine the software's source code undermined the constitutional right to a fair trial and procedural fairness.

Inferred additional harm: Over a million defendants assessed by COMPAS since 1998 may have had their civil rights compromised due to the opaque nature of the risk scoring.

People affected

• Occurrences reported: 1
• People reportedly harmed: 3
• People reportedly exposed: 1000000

Potential causes

Management

• Commercial Profit Prioritization: Companies prioritized proprietary interests over transparency and justice.
• Inadequate Risk Mitigation: Developers failed to proactively address bias and accuracy limitations.

Technology

• Opaque Black-Box Models: Proprietary algorithms prevent courts and defendants from verifying accuracy.
• Undetected Software Coding Errors: Bugs in software like STRmix and FST led to flawed outputs in court cases.
• Ineffective Prediction Accuracy: COMPAS performed no better than untrained online survey takers.

Data Inputs

• Biased Historical Arrest Data: Systemic bias in policing data led to racially disparate risk scores.
• Proxy Variables for Race: Factors like prior convictions act as proxies, embedding racial bias.
• Unrepresentative Training Data: Algorithms trained on non-Indigenous groups caused flawed risk assessments.

Human Factors

• Cognitive Anchoring Bias: Judges and officers over-rely on risk scores without examining flaws.
• Lack of Algorithmic Literacy: Judges did not understand how risk scores were mathematically computed.

Process and Methods

• Group-Based Analytics Used: Use of group averages deprives defendants of individualized sentencing.
• Lack of Source Code Audits: No independent adversarial testing was done on proprietary source codes.
• Self-Validation of Software: Developers co-authored validation studies, creating conflict of interest.

Regulatory Environment

• Trade Secret Protections: IP laws allowed companies to shield source code from defense scrutiny.
• Lack of Regulatory Oversight: No standards require developers to prove algorithm accuracy before deployment.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide extensive, consistent, and peer-reviewed scientific evidence (from Dartmouth researchers published in Science Advances) and investigative journalism (ProPublica) detailing the accuracy, bias, and transparency issues of COMPAS and TrueAllele. The legal and social implications are well-documented across multiple high-quality sources.
• Ambiguities identified: There is some debate between the developers (Equivant) and researchers regarding the exact number of active predictors used in COMPAS (6 vs 137), but this does not affect the overall findings of equal accuracy to untrained humans.
• Alternative interpretations: The developers argue that matching the 0.70 AUC standard is a validation of the tool's design rather than a failure, interpreting the results as proof of the model's validity.

The widespread deployment of proprietary, biased algorithmic tools like COMPAS in the US judicial system has led to systemic racial discrimination and due process violations. While not a kinetic or foreign threat, the scale of civil rights compromise and the erosion of public trust in core democratic institutions represent a substantial domestic societal stability concern.

• Overall national security impact: Substantial
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The deployment and use of these tools represent an ongoing, systemic strategic concern rather than an immediate crisis.
• Autonomy: Human-controlled. The algorithms act as decision-support tools, with human judges and parole boards retaining final decision-making authority.
• Novelty: Established threat. The use of actuarial and algorithmic risk assessment tools in the justice system is an established practice with known historical precedents.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic attacks, or critical infrastructure compromise indicated in the incident details.
• Information security: Negligible. No significant intelligence compromise or information warfare operations indicated.
• Sovereignty: Minor. Proprietary algorithms impacted domestic judicial decision-making and due process, but within standard legal and constitutional frameworks rather than a national security crisis.
• Economic security: Negligible. No threats to strategic industries, economic stability, or national technological competitive advantage are reported.
• Societal stability: Substantial. Systemic racial bias and due process violations in risk-assessment tools used across the US justice system represent a substantial, large-scale compromise of civil liberties.