Pennsylvania County's Family Screening Tool Allegedly Exhibited Discriminatory Effects

Data analysis by the American Civil Liberty Union (ACLU) on Allegheny County's decision-support Family Screening Tool to predict child abuse or neglect risk found the tool resulting in higher screen-in rates for Black families and higher risk scores for households with disabled residents.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The AFST algorithm systematically discriminates against Black and disabled families by assigning them higher risk scores, leading to unfair outcomes such as disproportionate child welfare investigations.

Additional risk subdomains

• 7.4 Lack of transparency or interpretability: The tool's risk scores and underlying variables are opaque, leaving affected families unable to understand, question, or contest the decisions made about them.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The discriminatory outcomes and biased risk scores were caused by the AFST AI system's predictions after it was deployed, though these risks were unintentional side effects of its design.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used by public authorities in a high-stakes social services context (child welfare risk assessment) which has significant implications for fundamental rights and safety.

AI system and alleged parties

• AI system: Allegheny Family Screening Tool 2.1
• AI purpose: Child Welfare Risk Assessment; Behavioral Modeling
• Behaviour type: Assistant
• Alleged developer: Rhema Vaithianathan, Emily Putnam-Hornstein, Centre for Social Data Analytics
• Alleged deployer: Allegheny County
• Alleged harmed parties: households with disabled people in Allegheny, Hackneys family, Black families in Allegheny

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Substantial
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Minor
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: Yes, the reports explicitly describe systemic differential treatment of Black families and households with disabled residents.

Directly caused: The AFST algorithm directly assigned higher risk scores to households with disabled individuals and Black families due to the use of biased administrative data.

Indirectly caused: This led to disproportionate child welfare investigations and family separations for these demographic groups, as seen in the Hackney family's case.

Inferred additional harm: Hundreds of Black and disabled families in Allegheny County have likely experienced disproportionate surveillance and intervention by child protective services due to the tool's biased scoring.

Civil rights

Reported: Yes, the reports describe violations of civil rights, specifically discrimination based on disability and race, and unwarranted family separations.

Indirectly caused: The Hackney family's right to family integrity was violated when their child was removed, influenced by the AFST. The civil rights of Black and disabled families are violated through systemic discriminatory screening practices.

Inferred additional harm: Numerous families have had their civil rights (protection against discrimination, family integrity) violated due to the county's reliance on this discriminatory algorithm.

Privacy

Reported: Yes, the reports describe child welfare workers searching homes, closets, and taking pictures of undressed children without warrants, driven by algorithmic screening.

Indirectly caused: Families subjected to investigations triggered by the AFST suffered severe privacy violations, including warrantless home searches and invasive personal questioning.

Inferred additional harm: Many families in the county have had their privacy rights violated through invasive investigations prompted by the tool.

Psychological

Reported: Yes, the reports explicitly describe severe emotional distress and trauma experienced by the Hackney parents due to the removal of their child.

Indirectly caused: The Hackney parents (2 people) suffered severe emotional distress and trauma from having their daughter removed from their custody, an action influenced by the AFST's risk assessment.

Inferred additional harm: It is highly likely that hundreds of other families subjected to unwarranted child welfare investigations and family separations driven by the tool's biased scores experienced similar psychological trauma.

People affected

• Occurrences reported: 1
• People reportedly harmed: 3
• People reportedly exposed: 3

Potential causes

Management

• Ignoring Research Feedback: County and developers ignored independent critical audits of the tool.
• Prioritizing Efficiency: Focusing on screening speed and cash-strapped budgets over equity.
• Lack of System Transparency: Management refused to disclose individual scores or explain variables used.

Technology

• Misleading Risk Grouping: Tool displays only the highest score, treating families as risky by association.
• Arbitrary Feature Selection: Thresholds for selecting variables were based on trial and error.
• Algorithmic Opacity: The risk calculation details and scores are kept hidden from families.

Data Inputs

• Biased Historical Records: Juvenile justice and jail databases overrepresent marginalized groups.
• Disproportionate Health Data: Behavioral health data penalizes families seeking disability-related care.
• Socioeconomic Data Skew: Lack of private healthcare data overrepresents low-income families.

Human Factors

• Automation Bias: Screeners treat mathematical outputs as objective, unquestionable truths.
• Lack of Recourse: Affected families cannot access or challenge their assigned risk scores.
• Screener Information Deficit: Call screeners only see aggregated household labels rather than details.

Process and Methods

• Mandatory Screen-in Protocols: High scores automatically trigger mandatory investigations with little override.
• Unchangeable Past Factors: Using permanent historical records offers families no way to escape their past.
• Inadequate System Audits: The tool was deployed without addressing potential equity and bias impacts.

Regulatory Environment

• Lack of Algorithmic Standards: No federal standards exist to regulate AI tools in child welfare systems.
• Post-Hoc Enforcement: Federal civil rights agencies only investigate after harm has occurred.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide detailed, credible accounts from both the affected family (the Hackneys) and an independent audit by the ACLU. The technical design choices of the AFST (V2.1) and their discriminatory impacts are thoroughly documented, and the ongoing federal investigation provides additional external validation of the safety risks.
• Ambiguities identified: The exact risk score assigned to the Hackneys is unknown due to the proprietary/opaque nature of the tool, meaning their specific classification is inferred based on their demographic profile.
• Alternative interpretations: One could argue the child removal was entirely due to the hospital's referral and caseworker judgment rather than the AFST, though the tool's mandatory screening protocols for high scores suggest it played a significant role.

A predictive child welfare algorithm in Pennsylvania exhibits systemic racial and disability bias, leading to civil rights concerns and federal investigations. While representing a serious local civil rights issue, the incident has negligible direct national security implications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing domestic policy and algorithmic bias concern rather than an immediate national security crisis.
• Autonomy: Human-supervised. The algorithm suggests risk scores to assist human caseworkers who make the final decisions regarding child welfare investigations.
• Novelty: Established threat. Algorithmic bias and discrimination in public sector decision-making tools are well-documented, established issues.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic targeting, or critical infrastructure compromises occurred in this incident.
• Information security: Negligible. No intelligence compromise, classified model theft, or information warfare operations are indicated.
• Sovereignty: Minor. Disrupts local county-level administrative decisions, but does not threaten national sovereignty or federal government functions.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or critical supply chain disruptions occurred.
• Societal stability: Minor. Causes localized civil rights concerns and systematic bias against marginalized groups, but lacks the population-scale destabilization required for a higher national security rating.