Department for Work and Pensions (DWP) Algorithm Wrongly Flags 200,000 for Housing Benefit Fraud

A Department for Work and Pensions (DWP) algorithm wrongly flagged over 200,000 UK housing benefit claims as high risk, resulting in unnecessary investigations. Two-thirds of these flagged claims were legitimate, causing wasted public funds and stress for claimants. Despite initial success in a pilot, the algorithm's real-world performance fell short. This incident highlights the risks of overreliance on automated systems in welfare administration.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The algorithm profiles individuals and determines fraud risk by weighing sensitive personal characteristics such as age and gender, leading to disproportionate and unfair targeting of vulnerable groups.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The algorithm's predictive accuracy was nearly half of what was expected, resulting in a 66% error rate and 200,000 wrongful investigations.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The high rate of false positives and subsequent wrongful investigations was an unexpected and unintended outcome of deploying the automated fraud detection system.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used by public authorities to evaluate eligibility for public assistance benefits, which is classified as high risk due to its significant impact on fundamental rights.

AI system and alleged parties

• AI system: DWP automated housing benefit fraud detection system
• AI purpose: Trustworthiness Scoring; Threat Detection
• Behaviour type: Tool
• Alleged developer: Department for Work and Pensions (DWP)
• Alleged deployer: Department for Work and Pensions (DWP)
• Alleged harmed parties: UK housing benefit claimants, UK general public

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Substantial, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Negligible
• Civil rights: direct Severe, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Substantial, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes. The reports state that about £4.4 million was spent on officials carrying out checks that did not save any money due to the algorithm's errors.

Directly caused: The UK government incurred a direct financial loss of approximately £4.4 million in wasted administrative costs for conducting fruitless reviews.

Inferred additional harm: Affected claimants may have faced indirect financial costs, such as communication expenses or temporary benefit disruptions, during the investigation process.

Differential treatment

Reported: Yes. The reports describe how the algorithm weighs personal characteristics like age and gender, leading to concerns of bias against vulnerable groups.

Directly caused: The system directly subjected over 200,000 legitimate claimants to unwarranted scrutiny based on demographic profiling.

Civil rights

Reported: Yes. Campaigners stated that the DWP's overreliance on the technology puts the rights of disadvantaged, marginalised, and vulnerable people in the backseat.

Directly caused: The civil rights of over 200,000 individuals were compromised by being subjected to invasive, automated profiling and unjustified investigations.

Privacy

Reported: Yes. Campaigners raised concerns about the DWP's relentless pursuit of privacy-invading technology and secretive mass profiling.

Directly caused: The system conducted invasive profiling of claimants' personal characteristics and tenancy agreements without transparent oversight.

People affected

• Occurrences reported: 1
• People reportedly harmed: 200000
• People reportedly exposed: 200000

Potential causes

Management

• Prioritizing Detection Over Rights: Focus on algorithm-led detection put vulnerable claimant rights last.
• Premature System Expansion: Expanded AI deployment despite warnings of algorithmic bias.

Technology

• Inaccurate Risk Prediction Algorithm: The algorithm was only 34-37 percent accurate compared to 64 percent in pilot.
• Algorithmic Bias: Risk of bias against vulnerable groups with protected characteristics.
• Invasive Profiling Tool: Mass profiling and citizen scoring based on personal characteristics.

Data Inputs

• Flawed Profiling Criteria: Weighed personal data like age, gender, and number of children.
• Discrepancy in Pilot Data: Pilot data showed 64 percent accuracy which failed to replicate in practice.

Human Factors

• Overreliance on Automation: DWP prioritized automated flags leading to unnecessary investigations.
• Lack of User-Centric Design: Failed to work with actual benefit claimants and users during development.

Process and Methods

• Inadequate Validation Processes: Inefficient manual reviews of flagged cases costing 4.4 million pounds.
• Lack of Transparency: Secretive algorithms used without public disclosure or clear explanations.

Regulatory Environment

• Lack of Independent Oversight: Inquiries failed to identify claimant harms despite high error rates.
• Insufficient Legislative Safeguards: Attempts to push intrusive mass monitoring bills without enough protection.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, official figures obtained via Freedom of Information requests, clearly detailing the scale of the incident, the financial impact, and the demographic inputs used.
• Ambiguities identified: The reports note that the specific housing benefit tool does not use machine learning, but discuss it alongside the DWP's broader deployment of AI, creating minor ambiguity about the technical complexity of the tool itself.
• Alternative interpretations: The incident could be interpreted as a standard bureaucratic or administrative failure rather than an AI safety incident, given the tool's lack of advanced machine learning capabilities.

The UK Department for Work and Pensions deployed an automated fraud-detection system that incorrectly flagged over 200,000 legitimate housing benefit claims. This resulted in 4.4 million pounds of wasted public funds on fruitless investigations and raised significant concerns about algorithmic bias and civil rights, representing a minor domestic governance and societal impact.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United Kingdom
• Alleged perpetrator: UK Department for Work and Pensions

Threat characteristics

• Imminence: Long-term. Represents an ongoing administrative and policy challenge rather than an immediate national security crisis.
• Autonomy: Human-supervised. The algorithm flagged claims autonomously, but human DWP officials made the final decisions to investigate and review the claims.
• Novelty: Established threat. Algorithmic bias and high false-positive rates in automated government decision-making are well-documented issues.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety were threatened by this administrative automated system.
• Information security: Negligible. No intelligence capabilities, classified information, or information warfare operations were involved in this incident.
• Sovereignty: Minor. The system wasted 4.4 million pounds in public funds and caused administrative inefficiency within a government department, but did not threaten state authority.
• Economic security: Minor. Caused minor economic loss of 4.4 million pounds in wasted administrative costs, but poses no threat to strategic industries or overall economic stability.
• Societal stability: Minor. Targeted over 200,000 legitimate claimants based on demographic profiling, raising human rights and bias concerns, but did not threaten national-scale societal stability.