Court Rules Deliveroo Used 'Discriminatory' Algorithm

In December 2020, an Italian court ruled that Deliveroo’s employee ‘reliability’ algorithm illegally discriminated against workers with legitimate reasons for cancelling shifts.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The algorithm discriminated against riders by penalizing them for exercising legally protected rights like striking or taking sick leave, leading to unfair work distribution.

Additional risk subdomains

• 6.2 Increased inequality and decline in employment quality: The algorithm was used to micromanage gig workers, reducing their employment quality and creating exploitative dependencies.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The discriminatory outcomes were caused by the automated decisions of the deployed Deliveroo ranking algorithm, which unintentionally penalized riders for legally protected absences.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is an employment and worker management system used to manage and evaluate gig workers. According to the definitions, High Risk includes 'Employment and worker management systems, such as AI used in recruitment' and managing workforces.

AI system and alleged parties

• AI system: Deliveroo reliability algorithm
• AI purpose: Workforce Monitoring and Evaluation; Scheduling
• Behaviour type: Autonomous
• Alleged developer: Deliveroo
• Alleged deployer: Deliveroo
• Alleged harmed parties: Deliveroo workers with legitimate reasons for cancelling shifts, Deliveroo workers

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Minor, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes, the report explicitly describes a court order for Deliveroo to pay 50,000 Euros to the applicants, plus legal costs, due to the discriminatory algorithm.

Directly caused: Deliveroo was ordered to pay 50,000 Euros (approximately 61,400 USD) in damages to the suing parties for the single occurrence of this systemic issue.

Indirectly caused: Riders who were penalized by the algorithm suffered indirect financial losses due to receiving fewer shift opportunities in busier, more lucrative timeblocks.

Inferred additional harm: It is highly likely that many individual riders suffered unquantified wage losses due to being deprioritized for shifts by the algorithm, though the exact total is not detailed in the reports.

Differential treatment

Reported: Yes, the report explicitly describes how the algorithm discriminated against riders who cancelled shifts for legally protected reasons.

Directly caused: The algorithm directly penalized riders who cancelled shifts due to illness or striking by lowering their reliability index, treating them the same as those who cancelled for trivial reasons.

Indirectly caused: This penalty resulted in these riders being deprioritized for future shift bookings, leading to unequal work opportunities.

Inferred additional harm: It is likely that hundreds of riders who fell ill or participated in strikes were systematically deprioritized and received fewer work opportunities compared to their peers.

Civil rights

Reported: Yes, the report describes a breach of local labor laws and the right to strike.

Directly caused: The algorithm penalized riders for exercising their legally protected right to strike and right to sick leave, which the court ruled as a violation of labor rights.

Inferred additional harm: The automated penalization likely discouraged riders from exercising their fundamental labor rights, such as striking, out of fear of losing future work.

People affected

• Occurrences reported: 1
• People reportedly harmed: 3
• People reportedly exposed: 3

Potential causes

Management

• Prioritizing Efficiency: Management prioritized supply consistency over labor rights.
• Inadequate Risk Assessment: Failed to evaluate potential legal liabilities of automated decisions.

Technology

• Flawed Ranking Algorithm: Algorithm did not distinguish between protected and trivial cancellations.
• Opaque Reliability Index: Blackbox system calculated metrics without transparent logic.

Data Inputs

• Incomplete Absence Data: System only recorded absence, ignoring reasons like sickness or strikes.

Human Factors

• Lack of Worker Recourse: Couriers could not appeal automated decisions or explain absences.

Process and Methods

• No Algorithmic Auditing: Absence of systematic reviews to identify indirect bias in algorithms.
• Discriminatory Shift Allocation: Booking system restricted shifts based on biased reliability metrics.

Regulatory Environment

• Evolving Gig Work Laws: Unclear regulations on algorithmic management in the self-employed sector.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, consistent details about the court ruling, the specific mechanics of the algorithm (penalizing cancellations within 24 hours), the parties involved, and the legal outcomes. There is minimal ambiguity regarding the AI's role or the nature of the harm.
• Ambiguities identified: The exact number of individual riders who were directly penalized by the system is not specified, only that the lawsuit was brought by three unions representing riders.
• Alternative interpretations: None. The court's legal ruling explicitly establishes the discriminatory nature of the algorithm under Italian labor law.

An Italian court ruled that Deliveroo's shift-allocation algorithm was discriminatory because it penalized riders for taking sick leave or striking. While representing an important legal precedent for gig-worker rights and algorithmic transparency, the national security implications of this commercial labor dispute are negligible.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: Italy
• Alleged perpetrator: Deliveroo

Threat characteristics

• Imminence: Long-term. The incident represents a regulatory and legal concern regarding labor practices rather than an active national security crisis.
• Autonomy: Full autonomy. The algorithm operated autonomously to calculate reliability scores and allocate shifts without real-time human intervention.
• Novelty: Evolved capability. While algorithmic management is common, using automated systems to penalize protected labor actions represents a significant evolution in algorithmic workplace discrimination.

Impact by dimension

• Physical security: Negligible. No threat to physical systems, critical infrastructure, or human safety was indicated in the incident reports.
• Information security: Negligible. No elements of information warfare, intelligence compromise, or systematic state-sponsored disinformation were present.
• Sovereignty: Negligible. The incident is a labor dispute involving a private company and does not impact government decision-making, elections, or territorial control.
• Economic security: Negligible. While involving labor rights in the gig economy, the economic impact is localized to a private firm and does not threaten strategic national industries or economic stability.
• Societal stability: Minor. The algorithm infringed on workers' rights to strike and take sick leave, representing a minor localized impact on labor rights, but did not cause widespread societal instability.