How French welfare services are creating ‘robo-debt’

A French welfare office using software to automatically evaluate cases incorrectly notified a woman receiving benefits that she owed €542.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The automated software failed to perform reliably when processing a file containing both salaried and freelance income, leading to an erroneous debt calculation.

Additional risk subdomains

• 5.2 Loss of human agency and autonomy: The system automatically initiated debt collection and scheduled deductions without prior human review or verification.
• 1.3 Unequal performance across groups: The data mining risk factors used by the system disproportionately target and impact vulnerable groups such as single women.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by an error in the automated welfare software's processing of a complex file, which was an unexpected outcome of its deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used to determine eligibility and calculate social welfare benefits, which is an essential public service with significant implications for fundamental rights.

AI system and alleged parties

• AI system: welfare office software
• AI purpose: Financial Processing; Resource Allocation
• Behaviour type: Autonomous
• Alleged developer: unknown
• Alleged deployer: French Welfare Offices
• Alleged harmed parties: Lucie Inland

Harm severity

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes potential differential treatment of individuals or groups caused directly or indirectly by the incident.

Directly caused: The system targeted the recipient, a single woman, based on automated 'risk factors' like starting freelance work.

Indirectly caused: N/A

Inferred additional harm: Systemic differential treatment of single women, foreigners, and those with changing job situations who are disproportionately targeted for welfare audits and debt collection.

Civil rights

Reported: The report explicitly describes potential violations of human or civil rights caused directly by the incident.

Directly caused: The recipient's benefits were altered and debt collection initiated by an automated process without prior human verification or clear notification of algorithmic use, violating GDPR Article 22 and French transparency laws.

Indirectly caused: N/A

Inferred additional harm: Widespread violations of GDPR Article 22 and French algorithmic transparency laws across the French welfare system affecting thousands of recipients.

Psychological

Reported: The report explicitly describes psychological distress and anxiety caused directly by the incident.

Directly caused: The recipient experienced significant distress, describing the situation as an 'emotional roller-coaster' and expressing 'rage' and feeling put in a 'difficult position'.

Indirectly caused: N/A

Inferred additional harm: It is highly likely that other welfare recipients subjected to incorrect automated debt notifications and deductions experienced similar psychological distress and anxiety.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Over-reliance on Data Mining: Management used automated data mining for three out of four welfare controls.
• Inadequate Risk Assessment: Management failed to assess the financial impact of false positive flags.

Technology

• Flawed Algorithmic Logic: The software ignored salaried work and only processed freelance status.
• Rigid System Parameters: The system completely reset the user's file when freelance status was added.

Data Inputs

• Incomplete Data Processing: The system failed to combine salaried and freelance income streams.
• Biased Risk Factors: Risk factors disproportionately flag complex but legitimate situations.

Human Factors

• Lack of Human-in-the-Loop: The system issued debt notifications before any human verified the error.

Process and Methods

• Automated Debt Enforcement: Debt was automatically declared and scheduled for deduction without review.
• Lack of Algorithmic Disclosure: The notification email failed to state that the decision was automated.

Regulatory Environment

• GDPR Article 22 Non-Compliance: Automated decisions with legal impact were made without proper safeguards.
• Weak Regulatory Enforcement: The data protection authority failed to intervene or provide transparency.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides a clear, first-hand account of the automated welfare system's failure, the specific financial amounts involved, and the caseworker's confirmation of the software's role. Sociological context further supports the systemic nature of the risk.
• Ambiguities identified: The exact technical parameters and algorithms used by the 'datamining' software are not fully detailed.
• Alternative interpretations: The incident could be interpreted as a human administrative error in setting up the software parameters, but the caseworker confirmed the software automatically analyzed and reset the file.

An automated French welfare software incorrectly calculated a 542 euro debt for a recipient due to a system error in processing mixed income types. While the incident highlights challenges with automated public decision-making and potential GDPR compliance issues, it represents a minor administrative failure with negligible national security implications.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Single nation
• Primary target: France
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The specific incident is resolved, representing an ongoing policy and regulatory discussion rather than an active crisis.
• Autonomy: Human-supervised. The software automatically scheduled deductions, but human caseworkers retained the capability to review, intervene, and reverse the decision.
• Novelty: Established threat. Algorithmic errors and database mismatches in public administration are well-established issues with existing administrative remedies.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety elements were affected or threatened by this administrative software error.
• Information security: Negligible. The incident involves domestic welfare administration and does not feature intelligence compromise, espionage, or information warfare.
• Sovereignty: Negligible. While the software is used in public service, a single administrative billing error does not threaten state authority, elections, or core government operations.
• Economic security: Negligible. No strategic technologies were compromised, and the minor financial discrepancy of 542 euros has no national economic or technological security implications.
• Societal stability: Minor. The automated system caused individual distress and raised minor civil rights concerns regarding automated decisions under GDPR, but impact was limited and resolved by a caseworker.