UK Government AI Allegedly Targets Disproportionate Numbers of Certain Nationals for Fraud Review

The UK's Department for Work and Pensions (DWP) faced scrutiny after many Bulgarian nationals reported unexplained suspensions of their Universal Credit benefits. The MP for Edmonton raised concerns about potential nationality-based targeting for benefit fraud investigations, leading to poverty and homelessness among affected individuals. The Home Office's own equality impact assessment found it was flagging a disproportionate number of marriages from Greece, Albania, Bulgaria and Romania.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The DWP's IRIS algorithm and the Met Police's facial recognition system exhibited unequal performance and accuracy rates across different demographic and national groups.

Additional risk subdomains

• 7.4 Lack of transparency or interpretability: The DWP consistently refused to disclose details about the algorithm's training data, functionality, or equality impact assessments.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the post-deployment operation of the IRIS algorithm, which unintentionally produced biased outcomes and led to the suspension of legitimate claims.

EU AI Act risk tier

• Risk tier: 2 High Risk

Risk Level 2: High Risk. The systems described include law enforcement facial recognition and public welfare eligibility evaluation, both of which have significant implications for fundamental rights.

AI system and alleged parties

• AI system: DWP Integrated Risk and Intelligence Service (IRIS) algorithm
• AI purpose: Application Evaluation; Threat Detection
• Behaviour type: Assistant
• Alleged developer: Home Office, Department for Work and Pensions, British government
• Alleged deployer: Various British government offices, Home Office, Department for Work and Pensions, British government
• Alleged harmed parties: Romanians in the United Kingdom, Greeks in the United Kingdom, Bulgarians in the United Kingdom, British public, Albanians in the United Kingdom

Harm severity

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Minor, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Minor, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Substantial
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: The reports describe claimants losing their Universal Credit benefits for months, being unable to pay rent, and facing eviction.

Directly caused: Average financial loss per occurrence is difficult to quantify precisely, but individual claimants lost months of Universal Credit payments, leading to severe rent arrears.

Indirectly caused: Claimants incurred indirect financial harms such as rent arrears, eviction costs, and reliance on food banks due to suspended benefits.

Inferred additional harm: We infer that thousands of pounds in benefits were wrongfully withheld from hundreds of vulnerable claimants, causing cumulative financial distress.

Differential treatment

Reported: The reports explicitly describe algorithmic bias and disproportionate targeting of specific nationalities and demographics.

Directly caused: The DWP algorithm disproportionately flagged Bulgarian nationals, while the Home Office sham marriage tool disproportionately targeted individuals from Albania, Greece, Romania, and Bulgaria.

Indirectly caused: The Metropolitan Police's facial recognition software falsely detected black people at a rate five times higher than white people under certain sensitivity settings.

Inferred additional harm: It is highly likely that systemic bias in these public sector algorithms led to widespread, unrecorded discriminatory outcomes against ethnic minorities and foreign nationals.

Civil rights

Reported: The reports describe concerns about violations of fundamental rights, lack of transparency, and the right to appeal.

Directly caused: The suspension of benefits without explanation or a clear route of appeal violated claimants' rights to fair administrative process and social security.

Indirectly caused: The use of biased facial recognition cameras by the police represents an intrusive surveillance practice that threatens civil liberties and privacy rights.

Inferred additional harm: Widespread deployment of opaque algorithms across multiple government departments likely infringed upon the civil rights of thousands of citizens without their knowledge.

Privacy

Reported: Campaigners raised concerns about privacy invasions and lack of data protection transparency.

Directly caused: The DWP's automated data analytics and profiling of historical claimant data to predict fraud risk constitutes an invasion of claimants' privacy.

Indirectly caused: The Metropolitan Police's live facial recognition cameras represent an intrusive form of public surveillance violating general privacy expectations.

Inferred additional harm: The systematic ingestion and processing of personal data across eight government departments likely resulted in widespread, unconsented privacy intrusions.

Psychological

Reported: The reports describe constituents being left in 'destitution' and facing eviction, which causes severe distress and anxiety.

Directly caused: N/A

Indirectly caused: Dozens of claimants experienced severe distress, anxiety, and the trauma of destitution and potential eviction due to unexplained benefit suspensions.

Inferred additional harm: It is likely that hundreds of other affected claimants suffered significant psychological distress and anxiety due to sudden, unexplained loss of income.

People affected

• Occurrences reported: 1
• People reportedly harmed: 100
• People reportedly exposed: 5900000

Potential causes

Management

• Prioritizing speed over safety: Pressure to speed up AI development and generate savings of 1.6 billion.
• Culture of secrecy: Management blocks FOI requests and MPs questions to protect capabilities.
• Justifying indirect discrimination: Home Office equality assessment justified discrimination as being proportionate.

Technology

• Self-learning algorithmic bias: Self-learning models balance data in opaque ways leading to unintended bias.
• Sensitivity setting errors: Lowering facial recognition sensitivity multiplied errors for black faces.
• Opaque triage algorithms: Algorithms flag claims or marriages for review with high false positive rates.

Data Inputs

• Historical fraud data bias: Models trained on historical data replicate past systemic biases.
• Incomplete demographic data: Lack of optional claimant demographic data limits bias testing capability.
• Segregated security data: Segregation of personal data prevents comprehensive fairness analysis.

Human Factors

• Caseworker over-reliance: Caseworkers suspend benefits based on algorithmic flags without clear reasons.
• Lack of claimant understanding: Vulnerable claimants cannot understand or challenge automated decisions.
• Lack of manual check diligence: Human intervention failed to prevent unfair benefit suspensions.

Process and Methods

• Lack of transparency: DWP refuses to publish information on how AI tools operate.
• Inadequate bias testing: Limited capability to test for unfair impacts across protected characteristics.
• Deficient appeal mechanisms: Claimants notified of suspensions by text without appeal information.

Regulatory Environment

• Voluntary disclosure database: Cabinet Office database is voluntary, letting departments hide AI tools.
• Lack of independent oversight: No outside body is handed an oversight role to protect fundamental rights.
• Abuse of FOI exemptions: Departments use FOI exemptions to avoid publishing details of AI tools.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the DWP's IRIS algorithm, its funding, its impact on Bulgarian nationals, and the broader context of public sector AI use in the UK. Multiple independent sources, including the National Audit Office and MPs, corroborate the findings.
• Ambiguities identified: The exact technical mechanism by which the self-learning algorithm developed bias against Bulgarian nationals is not fully explained, as the DWP refused to disclose the model details.
• Alternative interpretations: The DWP argues that the suspensions were part of standard risk reviews and not directly caused by algorithmic nationality bias, suggesting human caseworkers made the final decisions.

The UK government's deployment of machine learning algorithms for welfare fraud detection, immigration screening, and police facial recognition resulted in systemic bias and discriminatory outcomes against ethnic minorities and foreign nationals. While not a kinetic threat, the incident highlights significant civil rights, privacy, and societal stability concerns arising from opaque public sector AI integration.

• Overall national security impact: Substantial
• Response level: Substantial
• Scope: Single nation
• Primary target: United Kingdom
• Alleged perpetrator: UK Department for Work and Pensions

Threat characteristics

• Imminence: Long-term. Represents an ongoing policy and technological governance challenge rather than an immediate crisis requiring urgent national security intervention.
• Autonomy: Human-supervised. The algorithms assist decision-making by flagging cases for human review, though systemic failures occurred due to a lack of meaningful human intervention.
• Novelty: Evolved capability. Algorithmic bias is an established issue, but its coordinated scaling across multiple core government sectors represents an evolved systemic risk.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic attacks, or critical infrastructure compromise were reported in connection with this incident.
• Information security: Negligible. No information warfare, deepfake operations targeting institutions, or intelligence security compromises were indicated.
• Sovereignty: Minor. Internal government decision-making for welfare and immigration was affected by biased algorithms, but this did not threaten state sovereignty or core constitutional processes.
• Economic security: Negligible. Financial hardships were confined to individual welfare claimants and did not pose a systemic threat to the nation's economic or technological security.
• Societal stability: Substantial. Systemic algorithmic bias across welfare, immigration, and policing led to discriminatory outcomes against specific nationalities and demographic groups, raising notable civil rights and public surveillance concerns.