Airbnb's Trustworthiness Algorithm Allegedly Banned Users without Explanation, and Discriminated against Sex Workers

Airbnb allegedly considered publicly available data on users to gauge their trustworthiness via algorithmic assessment of personality and behavioral traits, resulting in unexplained bans and discriminatory bans against sex workers.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The algorithm systematically targets and excludes individuals based on their occupation, such as sex workers, leading to unfair discrimination and denial of service.

Additional risk subdomains

• 5.2 Loss of human agency and autonomy: Users are banned by an automated algorithm with no explanation and no meaningful avenue for human review or appeal.
• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: The algorithm secretly harvests and analyzes users' and their friends' social media data, employment history, and other personal information to infer sensitive traits.

Causal factors

• Entity: AI
• Intent: Intentional
• Timing: Post-deployment

The risk is caused by the automated decisions of Airbnb's trustworthiness algorithm, which was intentionally deployed to profile and exclude users post-deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used for automated decision-making that significantly impacts fundamental rights and access to essential services like housing, which aligns with high-risk classifications under the EU AI Act.

AI system and alleged parties

• AI system: Airbnb trustworthiness algorithm
• AI purpose: Trustworthiness Scoring; Application Evaluation
• Behaviour type: Autonomous
• Alleged developer: Trooly, Airbnb
• Alleged deployer: Airbnb
• Alleged harmed parties: sex workers, Airbnb users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Severe, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: Yes, the reports explicitly describe sex workers and people associated with the adult industry being systematically targeted and banned based on their occupation.

Directly caused: Sex workers (such as the erotic masseur and the female sex worker) were directly banned from the platform based on their occupation and online ads.

Inferred additional harm: It is highly likely that many other sex workers and individuals from marginalized groups or specific occupations have been systematically excluded from using Airbnb due to the algorithm's biased profiling.

Privacy

Reported: Yes, the reports describe Airbnb secretly collecting and analyzing users' personal data, social media accounts, and friends' networks without explicit consent or transparency.

Directly caused: The privacy of millions of Airbnb users was violated through cross-contextual data harvesting and profiling of their social networks, employment, and education.

Inferred additional harm: The continuous, opaque surveillance of users' online footprints likely constitutes a massive, ongoing violation of privacy rights for millions of global customers.

Psychological

Reported: Yes, the reports describe users feeling shocked, outraged, and devastated by the sudden, unexplained bans.

Directly caused: At least three individuals (Macheda, an erotic masseur, and a sex worker) experienced distress, outrage, and feeling devastated due to being banned without explanation.

Inferred additional harm: Many other users who were suddenly banned likely experienced similar distress and anxiety due to the lack of recourse and opaque process.

People affected

• Occurrences reported: 1
• People reportedly harmed: 4
• People reportedly exposed: 1000000

Potential causes

Management

• Prioritizing Security over Fairness: Focusing on platform risk mitigation at the expense of consumer rights.
• Acquisition of Trooly Startup: Integrating intrusive background-check tech without adequate oversight.
• Poor Algorithmic Governance: Implementing automated decisions without human oversight or auditing frameworks.

Technology

• Opaque Profiling Algorithm: Proprietary algorithm rates trustworthiness using opaque, unverified metrics.
• Cross-Contextual Analysis: Evaluates unrelated data like education and social media to predict behavior.
• Flawed Personality Assessment: Attempts to assess traits like narcissism using junk science social scoring.

Data Inputs

• Inaccurate Online Data: Algorithm relies on potentially incorrect social media and third-party data.
• Guilty by Association: Scores are adjusted based on the social media activity of friends.
• Discriminatory Data Points: Screens for occupation, old charges, and sensitive personal histories.

Human Factors

• Occupational Bias: Systemic bias against sex workers leads to unfair and discriminatory bans.
• Lack of User Agency: Users cannot opt out of profiling or check the accuracy of collected data.
• Inability to Self-Protect: Consumers cannot easily prevent data collection without going entirely offline.

Process and Methods

• No Appeals Process: Airbnb does not offer a clear mechanism to contest or review bans.
• Opaque Decision Communication: Banned users are given zero explanation or reasoning for their removal.
• Vague Privacy Policies: Policies use vague language about profiling, hiding how data is actually used.

Regulatory Environment

• Significant Regulatory Gap: Lack of strong laws protecting consumers from invasive automated profiling.
• Absence of Social Scoring Ban: No local frameworks equivalent to the EU ban on social scoring algorithms.
• Weak Enforcement of AI Rules: Failure to enforce established public policies like the OECD AI Principles.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts from consumer advocacy groups (Choice) and legal experts regarding Airbnb's acquisition of Trooly and the deployment of its trustworthiness profiling algorithm. The specific impacts, including unexplained bans of sex workers and other individuals, are well-documented across multiple sources, leaving little ambiguity about the system's function and consequences.
• Ambiguities identified: The exact weightings of the algorithm's scoring metrics and the degree of human involvement in the final ban decisions are not fully transparent due to Airbnb's vague public statements.
• Alternative interpretations: Airbnb claims these are standard security reviews designed to protect the community, rather than an invasive social credit system.

Airbnb deployed an automated trustworthiness profiling algorithm that harvests extensive personal and social media data to score and ban users. While raising notable civil rights, privacy, and discrimination concerns reminiscent of social credit systems, the incident remains a corporate regulatory and consumer protection issue with negligible direct national security impact.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: Australia
• Other affected: Global
• Alleged perpetrator: Airbnb

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic and regulatory concern regarding private data harvesting rather than an active, immediate national security crisis.
• Autonomy: Full autonomy. The algorithm automatically analyzes data, generates scores, and executes account bans without human intervention or transparent appeal processes.
• Novelty: Evolved capability. Represents an advanced, evolved application of consumer profiling and predictive behavioral scoring by a major tech platform.

Impact by dimension

• Physical security: Negligible. No physical security threat, kinetic attacks, or critical infrastructure compromise indicated in the incident details.
• Information security: Negligible. No evidence of state-sponsored intelligence compromise, classified data theft, or information warfare operations.
• Sovereignty: Negligible. The incident involves a private corporate platform's internal security screening and does not impact government decision-making, elections, or state sovereignty.
• Economic security: Negligible. Impact is limited to individual consumer access to a lodging platform, with no systemic threat to national financial systems or strategic technologies.
• Societal stability: Minor. Opaque profiling algorithm and data harvesting raise privacy and discrimination concerns for users, but impact is manageable under consumer protection and privacy regulations.