Oregon’s Screening Tool for Child Abuse Cases Discontinued Following Concerns of Racial Bias

Oregon’s Department of Human Services (DHS) stopped using its Safety at Screening Tool, that is aimed to predict the risk that children wind up in foster care or be investigated in the future, and opted for a new process allegedly to reduce disparities and improve racially equitable decision-making.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The algorithm exhibited unequal performance and predictive disparities across demographic groups, disproportionately flagging Black families for child neglect investigations.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The systemic over-flagging of Black families led to unfair outcomes and unequal treatment by child protective services.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The racial disparities and bias in the child welfare screening decisions were caused by the deployed AI algorithm, which was an unintentional and unexpected outcome of its deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used by public authorities in child welfare to make critical decisions about family intervention and child safety, which has significant implications for fundamental rights and public interests. The report notes that 'making decisions about what should happen to children and families is far too important a task to give untested algorithms'.

AI system and alleged parties

• AI system: Allegheny Family Screening Tool, Safety at Screening Tool
• AI purpose: Child Welfare Risk Assessment; Resource Allocation
• Behaviour type: Assistant
• Alleged developer: Oregon Department of Human Services
• Alleged deployer: Oregon Department of Human Services
• Alleged harmed parties: families of minority groups, children of minority groups

Harm severity

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Negligible
• Civil rights: direct Substantial, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment of Black families who were disproportionately flagged for neglect investigations.

Directly caused: The algorithm flagged a disproportionate number of Black children for mandatory neglect investigations compared to other groups.

Indirectly caused: N/A

Inferred additional harm: Numerous Black families in Oregon and Pennsylvania likely experienced systemic differential treatment and higher rates of unwarranted state intervention due to algorithmic bias.

Civil rights

Reported: The report explicitly describes concerns about racial bias and civil rights violations in the child welfare system.

Directly caused: The disproportionate targeting of Black families for child neglect investigations represents a violation of their civil rights to equal treatment.

Indirectly caused: N/A

Inferred additional harm: Systemic civil rights violations likely affected many minority families who faced unwarranted investigations and potential family separations driven by biased algorithmic scores.

People affected

• Occurrences reported: 1

Potential causes

Management

• Adoption of Flawed Models: Management adopted a model inspired by another state's biased tool.
• High Leadership Turnover: Agency prepared to hire its eighth director in six years, limiting oversight.

Technology

• Algorithmic Racial Bias: The algorithm flagged a disproportionate number of Black families.
• Untested Predictive Models: High-stakes decisions were given to untested screening algorithms.
• Reliability Concerns: There were concerns about the reliability of the predictive technology.

Data Inputs

• Inadequate Data: The reunification pilot was paused due to inadequate data.
• Biased Internal Data: The model drew from internal child welfare data which may reflect bias.

Human Factors

• Hardening of Human Bias: Social workers viewing scores could have their existing biases hardened.

Process and Methods

• Lack of System Transparency: There was a lack of transparency in how the algorithm made decisions.
• Flawed Screening Process: The screening process did not ensure racially equitable decisions.

Regulatory Environment

• Lack of National Oversight: Absence of federal regulations and oversight for child welfare algorithms.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly outlines the decision to discontinue the algorithm, the reasons (racial bias and disparities), and the context of similar tools in other states. While specific statistical data on the exact number of affected families is missing, the overall narrative and policy implications are well-documented.
• Ambiguities identified: The exact statistical extent of the racial disparities in Oregon's specific tool is not detailed, as the spokesman declined to elaborate on the specific disparities that influenced the policy change.
• Alternative interpretations: None. The discontinuation of the tool due to racial bias concerns is clear and uncontested.

The Oregon Department of Human Services discontinued its child welfare screening algorithm due to concerns over racial bias. While the incident highlights significant civil rights and societal concerns regarding algorithmic fairness in public administration, its national security impact remains negligible to minor, with no threats to physical security, sovereignty, or critical infrastructure.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The tool has been discontinued and the immediate issue resolved, representing a long-term strategic concern regarding algorithmic bias in governance.
• Autonomy: Human-supervised. The algorithm provided risk scores to assist social workers, who retained final decision-making authority and could override the tool.
• Novelty: Established threat. Algorithmic bias and predictive disparities in public sector decision-making are well-documented issues with existing precedents.

Impact by dimension

• Physical security: Negligible. No physical threat to critical infrastructure, weapons systems, or public safety was identified in this domestic administrative incident.
• Information security: Negligible. The incident involves a domestic public welfare algorithm and does not involve foreign intelligence operations or information warfare.
• Sovereignty: Negligible. The incident involves a local state government administrative tool and does not threaten national sovereignty or core federal government functions.
• Economic security: Negligible. No economic warfare, financial system threats, or strategic technology theft were associated with this child welfare tool.
• Societal stability: Minor. The algorithm disproportionately flagged Black families, raising civil rights concerns, but the impact was localized and managed through standard administrative procedures by discontinuing the tool.