Predictive Policing Biases of PredPol

Predictive policing algorithms meant to aid law enforcement by predicting future crime show signs of biased output.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The COMPAS algorithm exhibits unequal performance across demographic groups, falsely flagging Black defendants as high-risk at twice the rate of White defendants, while PredPol's accuracy and targeting disproportionately burden minority communities due to biased training data.

Additional risk subdomains

• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: PredPol left millions of sensitive crime predictions and location data exposed on an unsecured AWS cloud storage bucket, and IBM developed video analytics using secret access to public camera footage.
• 5.1 Overreliance and unsafe use: Judges and police departments over-relied on algorithmic risk scores and patrol maps to make critical sentencing and law enforcement decisions, despite the systems' inherent biases.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The systemic racial and socioeconomic biases and disproportionate targeting were caused by the outputs of the deployed AI systems, which was an unexpected and unintentional outcome of pursuing crime prevention goals.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes AI systems used in law enforcement and judicial applications, which have significant implications for fundamental rights, safety, and public interests.

AI system and alleged parties

• AI system: COMPAS, Intelligent Video Analytics, PredPol (IBM, Northpointe)
• AI purpose: Predictive Policing; Recidivism Prediction
• Behaviour type: Assistant
• Alleged developer: PredPol
• Alleged deployer: PredPol, Oakland Police Department
• Alleged harmed parties: Oakland Residents

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Substantial
• Civil rights: direct Substantial, indirect Substantial
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Severe, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: The report explicitly describes indirect financial losses resulting from arrests and housing instability.

Indirectly caused: A resident named [PERSON_016] faced eviction and had to move her family to a hotel, incurring immediate lodging and moving expenses due to her boyfriend's arrest in a targeted patrol area.

Inferred additional harm: Thousands of low-income individuals likely suffered significant cumulative financial losses from court fees, bail, lost wages, and legal representation due to biased arrests and harsher sentences driven by the algorithms.

Differential treatment

Reported: The report explicitly describes extensive differential treatment of racial minorities and low-income groups caused directly and indirectly by the systems.

Directly caused: PredPol's algorithm targeted Black, Latino, and low-income neighborhoods for increased patrols, while COMPAS falsely flagged Black defendants as high-risk at twice the rate of White defendants.

Indirectly caused: Minority and low-income individuals faced disproportionate police contact, higher arrest rates, and harsher sentencing compared to wealthier, White individuals.

Inferred additional harm: Systemic reinforcement of racial and socioeconomic biases across dozens of U.S. cities, affecting hundreds of thousands of residents over several years of deployment.

Civil rights

Reported: The report explicitly describes violations of civil rights and due process caused directly and indirectly by the incident.

Directly caused: The use of biased risk scores directly violated defendants' due process rights during sentencing, as seen in the case of Eric Loomis.

Indirectly caused: Unjust arrests, over-policing, and discriminatory law enforcement practices violated the civil liberties of residents in targeted communities.

Inferred additional harm: Widespread civil rights violations across the United States due to the unconstitutional and opaque use of biased algorithms in law enforcement and the judiciary.

Privacy

Reported: The report explicitly describes privacy violations caused directly by the exposure of sensitive data.

Directly caused: PredPol left over 7.8 million crime predictions and sensitive location data exposed on an unsecured AWS cloud storage bucket, accessible to the public without a password.

Inferred additional harm: Potential unauthorized access and exploitation of police patrol routes and targeted residential addresses, compromising the privacy of thousands of citizens.

Psychological

Reported: The report explicitly describes psychological distress and trauma caused indirectly by the over-policing and biased arrests.

Indirectly caused: An advocate in Birmingham noted that the high density of police presence reopens generational trauma and hurts communities. Additionally, a resident named [PERSON_016] broke down in tears upon receiving an eviction notice resulting from her boyfriend's arrest in a heavily policed area.

Inferred additional harm: Widespread anxiety, stress, and psychological trauma are highly likely among thousands of residents living in minority and low-income neighborhoods subjected to relentless, algorithmically driven police surveillance.

People affected

• Occurrences reported: 1
• People reportedly harmed: 2000
• People reportedly exposed: 10000000

Potential causes

Management

• Ignoring Bias Research: Management refused to update algorithms despite known racial disparities.
• Prioritizing Sales Over Safety: Selling software to agencies without enforcing proper usage guidelines.
• Lack of Client Training: Failed to ensure clients did not predict discouraged crime types.

Technology

• Feedback Loop Amplification: Algorithm targets areas with historical arrests, reinforcing existing patterns.
• Inaccurate Risk Predictions: COMPAS had high false positive rates for Black defendants.
• Mixing Unrelated Crime Types: Combining different crimes into single predictions reduced accuracy.

Data Inputs

• Biased Historical Crime Data: System relies on past arrests reflecting systemic policing biases.
• Use of Socioeconomic Proxies: Questionnaires use poverty and family history as proxies for crime risk.
• Incomplete Crime Reporting: Only a fraction of actual crimes are reported, biasing input data.

Human Factors

• Automation Bias by Judges: Judges rely heavily on risk scores to override plea deals.
• Officer Discretion and Abuse: Officers used predictions to justify stops and trespass arrests.
• Lack of Public Awareness: Local advocates and task forces were unaware of software deployment.

Process and Methods

• Lack of Model Transparency: Proprietary algorithms prevent public or defense scrutiny of scores.
• Inadequate Validation Studies: Jurisdictions deployed tools without rigorous local validation.
• Unsecured Data Storage: PredPol left millions of predictions exposed on an open AWS bucket.

Regulatory Environment

• Absence of Algorithmic Standards: No federal or state regulations govern predictive policing accuracy.
• Lack of Independent Oversight: No regulatory bodies audited the software for civil rights violations.
• Lack of Disclosure Mandates: Defense attorneys are not routinely informed when AI led to arrests.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide extensive, empirical, and peer-reviewed evidence detailing the systemic biases of PredPol and COMPAS, supported by data analysis of millions of predictions and court records. The findings are highly credible and well-documented.
• Ambiguities identified: The exact proprietary calculations and source code of COMPAS and PredPol remain undisclosed, limiting complete transparency.
• Alternative interpretations: Proponents argue the software merely reflects reported crime rates rather than introducing new bias, though the reports strongly refute this.

The deployment of biased predictive policing and recidivism algorithms across the United States has led to systemic civil rights violations and over-policing of minority communities. While it does not pose an immediate kinetic or foreign threat, the widespread erosion of trust in judicial and law enforcement institutions represents a substantial threat to domestic societal stability and human rights.

• Overall national security impact: Substantial
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The issue represents an ongoing, systemic domestic policy and civil rights concern rather than an immediate, acute national security crisis.
• Autonomy: Human-supervised. The AI systems generate risk scores and patrol recommendations, but human operators (police and judges) retain final decision-making authority.
• Novelty: Established threat. Algorithmic bias in predictive policing and risk assessments is a well-documented and established threat with numerous prior incidents.

Impact by dimension

• Physical security: Negligible. No physical attacks, autonomous weapons deployment, or critical infrastructure compromise reported in this incident.
• Information security: Minor. PredPol left millions of sensitive crime predictions exposed on an unsecured server, and IBM used unauthorized access to NYPD footage, representing minor data security and intelligence compromises.
• Sovereignty: Minor. Algorithmic bias impacted judicial sentencing and law enforcement operations, representing a minor disruption to constitutional due process and local government functions.
• Economic security: Negligible. No significant threats to financial systems, strategic technology theft, or loss of international technological advantage indicated.
• Societal stability: Substantial. Widespread use of biased algorithms led to systematic discrimination, over-policing of minority communities, and significant civil rights violations, posing substantial threats to societal stability.