Chicago Police's Strategic Subject List Reportedly Biased Along Racial Lines

Chicago Police Department (CPD)'s Strategic Subject List as output of an algorithm purportedly to identify victims or perpetrators of violence was reportedly ineffective, easily abused, and biased against low-income communities of color.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The SSL algorithm systematically assigned higher risk scores to Black and low-income residents, resulting in unfair profiling and discriminatory policing outcomes for these groups.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The system failed to perform effectively or reduce crime rates, as confirmed by independent RAND Corporation audits.
• 5.1 Overreliance and unsafe use: Police officers relied on the 'scientific' authority of the algorithm to justify targeted stops and arrests, laundering subjective biases into objective judgments.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The systemic racial profiling and feedback loops were unintentional and unexpected outcomes of deploying the predictive policing algorithm to allocate resources.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes an AI system used in law enforcement for profiling and predicting crimes, which has significant implications for safety, fundamental rights, and public interests.

AI system and alleged parties

• AI system: Strategic Subject List (Illinois Institute of Technology)
• AI purpose: Predictive Policing; Resource Allocation
• Behaviour type: Assistant
• Alleged developer: Chicago Police Department
• Alleged deployer: Chicago Police Department
• Alleged harmed parties: low-income communities, communities of color, Black Chicago residents

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Substantial
• Civil rights: direct Substantial, indirect Substantial
• Democracy: direct Minor, indirect Substantial
• Privacy: direct Substantial, indirect Severe
• Psychological: direct Minor, indirect Substantial
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes systematic differential treatment of Black and low-income residents caused directly by the incident.

Directly caused: The SSL algorithm disproportionately assigned high risk scores to Black residents, with 56% of young Black men in Chicago being placed on the list.

Indirectly caused: Police resources were disproportionately deployed to minority and low-income neighborhoods based on these biased scores, leading to increased street stops and arrests in already over-policed communities.

Inferred additional harm: Systemic racial profiling and unequal law enforcement treatment affecting hundreds of thousands of minority residents in Chicago and Oakland.

Civil rights

Reported: The report explicitly describes serious Fourth Amendment concerns regarding unreasonable searches, seizures, and profiling without reasonable suspicion.

Directly caused: Individuals were subjected to pre-emptive police visits, home monitoring, and targeted stops based on algorithmic scores rather than individualized reasonable suspicion.

Indirectly caused: The erosion of the presumption of innocence, as individuals were treated as 'criminals in waiting' or 'persons of interest' before committing any crime.

Inferred additional harm: Widespread civil rights violations under the Fourth Amendment for the 400,000 individuals placed on the list and subjected to heightened police scrutiny.

Democracy

Reported: The report explicitly describes a lack of transparency, public oversight, and accountability, with government decision-making turned over to secret algorithms.

Directly caused: The Chicago Police Department hid the SSL database and algorithm from the public, refusing independent audits and public records requests until forced by a lawsuit.

Indirectly caused: The use of 'black-box' machinery shifted accountability away from police chiefs and decision-makers, undermining public trust and democratic oversight of law enforcement.

Inferred additional harm: Long-term erosion of trust in public institutions and democratic governance as police departments increasingly rely on proprietary, un-auditable algorithms to make critical civic decisions.

Privacy

Reported: The report explicitly describes the collection and processing of vast amounts of personal data, including social media, arrest records, and social networks.

Directly caused: The SSL and Beware software processed billions of data points, including deep web searches, social media posts, and social network connections of individuals without their explicit consent.

Indirectly caused: The creation of a persistent surveillance state where individuals' associations and online activities were monitored to calculate 'threat scores.'

Inferred additional harm: Mass privacy violations affecting hundreds of thousands of citizens whose personal data and social connections were ingested and analyzed by predictive policing algorithms.

Psychological

Reported: The report explicitly notes that increased police scrutiny and surveillance have been linked to worsening mental and physical health, and describes individuals being taken by surprise by unannounced police visits.

Directly caused: Individuals placed on the SSL experienced surprise, distress, and anxiety from unannounced police visits and ongoing surveillance.

Indirectly caused: Widespread anxiety and feelings of being targeted among the young Black male population in Chicago, where 56% were placed on the list.

Inferred additional harm: Widespread psychological distress, anxiety, and trauma among the 400,000 individuals labeled as high-risk and their families due to persistent police surveillance and fear of wrongful arrest.

People affected

• Occurrences reported: 1
• People reportedly harmed: 400000
• People reportedly exposed: 400000

Potential causes

Management

• Prioritizing Tech over Strategy: Management adopts software as a panacea without clear operational plans.
• Failure to Release Internal Audits: Leadership keeps efficacy reports and algorithmic data shielded from review.
• Decommissioning without Long-term Plans: Programs like SSL are shut down due to poor planning and low efficacy.

Technology

• Feedback Loop Amplification: Algorithms reinforce prior biases by predicting crime where arrests occurred.
• Proprietary Black-Box Algorithms: Trade secrets prevent independent review of how variables are weighted.
• Inaccurate Prediction Models: Models like SSL had little to no impact on reducing actual crime rates.

Data Inputs

• Biased Historical Arrest Data: Models train on arrests reflecting past policing bias rather than actual crime.
• Use of Unconvicted Arrest Records: Systems use arrest records instead of convictions to calculate threat scores.
• Incomplete Crime Reporting: Less than half of violent crimes are reported, biasing the input data.

Human Factors

• Overreliance on Algorithm Outputs: Officers trust scores without understanding the underlying biases or weights.
• Lack of Officer Training: Field officers receive little training on how to interpret and use lists.
• Confirmation Bias in the Field: Officers patrol targeted areas expecting crime, confirming prior beliefs.

Process and Methods

• Lack of Public Transparency: Algorithms are deployed without public input or explanation of variables.
• Inadequate Intervention Protocols: Departments fail to pair risk lists with effective social interventions.
• Absence of Independent Audits: Very few independent evaluations of predictive policing tools are conducted.

Regulatory Environment

• Weak Fourth Amendment Protections: Predictive scores are used to bypass reasonable suspicion requirements.
• Absence of Algorithmic Standards: No regulatory standards govern the use of big data in law enforcement.
• Lack of Public Accountability: Legal barriers prevent citizens from challenging their inclusion on lists.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide extensive, detailed, and consistent accounts of the Chicago Police Department's Strategic Subject List (SSL), its development, deployment, and subsequent decommissioning. Independent audits by the RAND Corporation and investigative journalism provide empirical data on the system's scale (nearly 400,000 people scored) and its disproportionate impact on Black residents (56% of young Black men scored). The technical inputs and limitations of the algorithm are well-documented, leaving little ambiguity about the core facts of the incident.
• Ambiguities identified: The exact mathematical weights of the variables in the proprietary algorithm were kept secret by the CPD, though researchers successfully reverse-engineered them with high accuracy.

The Chicago Police Department deployed a predictive policing algorithm (SSL) that generated biased risk scores for 400,000 residents. While causing significant local civil rights concerns and systemic racial profiling, the incident had negligible direct impact on national security and was ultimately resolved through local oversight and decommissioning.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Chicago Police Department

Threat characteristics

• Imminence: Long-term. The specific system has been decommissioned, representing a historical case rather than an active, imminent national security crisis.
• Autonomy: Human-supervised. The AI assigned risk scores to assist police officers, who retained final decision-making authority regarding arrests and interventions.
• Novelty: Evolved capability. Represents an evolution of existing statistical and predictive policing methods scaled up through algorithmic automation.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic attacks, or critical infrastructure compromise occurred during the deployment of this predictive policing tool.
• Information security: Negligible. No evidence of foreign intelligence compromise, classified data theft, or systematic information warfare operations.
• Sovereignty: Negligible. Local law enforcement tool with no direct impact on national sovereignty, federal government decision-making, or territorial integrity.
• Economic security: Negligible. No threats to national financial systems, strategic technology theft, or loss of national technological competitive advantage.
• Societal stability: Minor. The tool caused localized civil rights concerns and systematic bias affecting 400,000 residents, but impacts were managed via standard legal and oversight procedures.