Facebook’s hate-speech detection algorithms was found by company researchers to have under-reported less common but more harmful content that was more often experienced by minority groups such as Black, Muslim, LGBTQ, and Jewish users.
.avif){kind=small}
Internal Facebook documents revealed that the company's hate speech detection algorithms were ineffective at removing severe, targeted hate speech against minority groups, while simultaneously over-indexing on less harmful content. Despite internal research identifying these failures and proposing an overhaul to better protect vulnerable groups, executives rejected the changes to maintain a 'politically neutral' appearance. This resulted in a platform environment where minority users, particularly Black users, faced disproportionate exposure to racist and derogatory language.
Facebook’s hate-speech detection algorithms was found by company researchers to have under-reported less common but more harmful content that was more often experienced by minority groups such as Black, Muslim, LGBTQ, and Jewish users.
The algorithm performed unequally across demographic groups, aggressively detecting comments denigrating White people and men while consistently failing to remove severe racist attacks against minority groups.
The incident stems from the unexpected failure and biased performance of Facebook's deployed hate speech detection algorithm, which did not perform as intended.
High Risk: The report describes an AI system with significant implications for safety, fundamental rights, and public interests, specifically violating civil rights and failing to protect vulnerable groups.
Severity ratings are based on this scale.
Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.
Reported: The report explicitly describes the spread of toxic and malicious content, including severe hate speech and slurs.
Directly caused: The algorithm failed to remove 95% to 97% of hate speech, leaving an estimated 250 million pieces of hate speech active in Groups.
Inferred additional harm: Widespread proliferation of toxic content across the entire platform, exposing millions of users to harmful language daily.
Reported: The report explicitly describes differential treatment of minority groups due to the algorithm's biased performance.
Directly caused: The algorithm disproportionately protected White users and men, with 90% of automated takedowns targeting content directed at them, while failing to protect vulnerable minorities.
Indirectly caused: Facebook executives rejected an overhaul to protect vulnerable groups, maintaining a system that left minorities more likely to encounter derogatory language.
Inferred additional harm: Systemic bias in content moderation algorithms across other platforms, leading to unequal protection for marginalized groups globally.
Reported: The report explicitly describes violations of civil rights, with independent auditors concluding Facebook's decisions were a tremendous setback for civil rights.
Directly caused: The failure to protect minority users from targeted harassment and hate speech violated their civil rights and restricted their freedom of expression.
Indirectly caused: Activists were silenced and forced off the platform due to safety concerns, undermining their ability to advocate for civil rights.
Inferred additional harm: Broad suppression of minority voices and civil rights advocacy on social media due to inadequate safety measures.
Reported: The report explicitly describes concerns about the erosion of democratic norms, specifically a chill factor in a democratic state.
Indirectly caused: The harassment of activists and the decline in minority participation on the platform undermines democratic discourse and representation.
Inferred additional harm: Large-scale erosion of public trust and democratic participation as marginalized communities are silenced or driven off major public squares.
Reported: The report explicitly describes psychological harm, including harassment, death threats, and a chill factor causing fear among users.
Directly caused: Activists like Malkia Devich-Black received relentless harassment and death threats, creating a chill factor and fear that forced them to stop managing pages.
Indirectly caused: Widespread distress and safety concerns among minority users, contributing to a 2.7% decline in Black monthly active users.
Inferred additional harm: Millions of minority users likely experienced psychological distress, anxiety, and feelings of unsafety due to unmoderated hate speech.
Reported: The report explicitly describes epistemic harm through Facebook's public deception regarding its proactive detection rates.
Directly caused: Facebook misled the public, Congress, and investors by touting a 90%+ proactive rate while hiding the dismal 3% to 5% actual takedown rate.
Inferred additional harm: Widespread public and regulatory misunderstanding of AI capabilities and the true scale of online hate speech.
Internal Facebook documents revealed systemic failures in hate speech detection algorithms, catching only 3 to 5 percent of hate speech while disproportionately failing to protect minority groups from harassment and death threats. Executives rejected proposed overhauls to maintain political neutrality, leading to significant civil rights setbacks and impacting societal stability.