Online Trolls Allegedly Abused TikTok’s Automated Content Reporting System to Discriminate against Marginalized Creators

TikTok's automated content reporting system was allegedly abused by online trolls to intentionally misreport content created by users of marginalized groups.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The automated moderation system disproportionately censored and banned a transgender Latina creator due to coordinated false reports, resulting in unfair outcomes and discrimination against marginalized groups.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI system failed to perform robustly against coordinated adversarial false reporting, leading to erroneous account deletions.
• 4.3 Fraud, scams, and targeted manipulation: Trolls exploited the automated reporting system to conduct targeted harassment and humiliation of a specific creator.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The automated moderation algorithm made the final decision to delete the account and deny the appeals, which was an unintended outcome of its design.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The system is an automated content moderation and spam filtering tool used in an entertainment application, which falls under Risk Level 4.

AI system and alleged parties

• AI system: TikTok automated content reporting system
• AI purpose: Content Moderation; NSFW Content Detection
• Behaviour type: Autonomous
• Alleged developer: TikTok
• Alleged deployer: TikTok
• Alleged harmed parties: TikTok content creators of marginalized groups

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Minor
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: The report explicitly describes financial losses, specifically the loss of funds accrued through the TikTok Creator Fund.

Directly caused: Rosalynne lost access to the funds she had already accrued in the TikTok Creator Fund due to her account being permanently deleted.

Indirectly caused: She lost her primary platform of 300,000+ followers, severely impacting her future earning potential and brand partnerships as a content creator.

Inferred additional harm: Other marginalized creators who had their accounts wrongfully deleted likely suffered similar losses of accrued and potential income.

Malicious content

Reported: The report explicitly describes toxic and malicious content spread by trolls targeting the creator.

Indirectly caused: The algorithm failed to remove videos from a troll who screen-recorded Rosalynne's content to spread hate, concluding they did not violate guidelines.

Inferred additional harm: The automated system's failure to moderate actual hate speech likely allowed toxic content targeting LGBTQPIA+ and BIPOC communities to proliferate.

Differential treatment

Reported: The report explicitly describes differential treatment of marginalized groups caused by the incident.

Directly caused: The automated moderation system censored and banned Rosalynne and other BIPOC, LGBTQPIA+, and disabled creators based on false reports, while failing to ban the trolls harassing them.

Indirectly caused: The platform's reliance on automated bans created an unequal environment where marginalized voices are disproportionately silenced.

Inferred additional harm: Systemic bias in automated moderation systems likely leads to widespread unequal treatment of marginalized creators across the platform.

Civil rights

Reported: The report explicitly describes violations of civil rights, specifically freedom of expression and non-discrimination.

Directly caused: Rosalynne's right to free expression and non-discrimination was violated when her educational content and account were deleted without a fair review.

Indirectly caused: The weaponization of the automated system by trolls effectively suppressed the civil rights of marginalized groups to participate in public discourse.

Inferred additional harm: Widespread automated censorship of marginalized groups represents a broader threat to digital civil rights and freedom of expression.

Psychological

Reported: The report explicitly describes psychological harm, including receiving hate comments and death threats from trolls.

Directly caused: Rosalynne experienced distress and frustration due to the sudden, automated deletion of her account and the rejection of her appeals.

Indirectly caused: The platform's failure to stop the trolls enabled them to continue harassing her, causing ongoing distress.

Inferred additional harm: Other marginalized creators targeted by similar mass-reporting campaigns likely suffered significant distress, anxiety, and feelings of exclusion.

People affected

• Occurrences reported: 1
• People reportedly harmed: 2
• People reportedly exposed: 100

Potential causes

Management

• Over-reliance on Automation: Management prioritized automated systems over human moderation teams.
• Lack of Creator Safeguards: Failure to implement safety measures protecting marginalized groups.

Technology

• Automated Report Threshold Trigger: Threshold-based automated removal allowed trolls to take down content.
• Flawed Moderation Algorithm: AI failed to distinguish transition updates from guideline violations.
• Inaccurate Livestream Detection: AI flagged benign clothing like tank tops as serious pornography.

Data Inputs

• Spam and False Report Signals: System treated coordinated malicious reports as valid data inputs.
• Unlabeled Harassment Re-posts: Re-posted videos used for harassment were not flagged by algorithms.

Human Factors

• Targeted Troll Harassment: Discriminatory trolls coordinated to abuse the automated reporting system.

Process and Methods

• Vague Community Guidelines: Platform guidelines are general, leading to inconsistent enforcement.
• Automated Appeal Review: Appeals are processed by computer programs instead of human moderators.
• Lack of Moderation Transparency: No clear reasons provided for content removal or denied appeals.

Information quality

• Classification confidence: Medium
• Reason for confidence: The report is a detailed first-hand account from an affected creator, providing clear evidence of the harm and the mechanism of the exploit. However, it lacks technical details about TikTok's internal moderation algorithms, and the exact role of human reviewers in the appeal process is inferred.
• Ambiguities identified: The exact threshold of reports required to trigger an automatic ban is unspecified, and the extent of human involvement in the denied appeals is unclear.
• Alternative interpretations: The bans could be interpreted as strict enforcement of platform guidelines rather than an algorithmic failure, though the evidence of targeted trolling makes this less likely.
• Missing information: Technical specifications of TikTok's automated moderation system, data on report-to-ban thresholds, and confirmation of whether human moderators reviewed the final account deletion.

Online trolls successfully exploited TikTok's automated moderation system to censor and ban a marginalized content creator. While demonstrating significant vulnerabilities in automated platform governance and algorithmic bias, the incident remains a localized platform abuse case with minor national security implications.

• Overall national security impact: Minor
• Scope: Unknown
• Primary target: No clear primary
• Alleged perpetrator: Online trolls

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing, systemic platform vulnerability rather than an active national security crisis requiring urgent response.
• Autonomy: Full autonomy. The moderation algorithm automatically executed account bans and content removals based on reporting thresholds without human oversight.
• Novelty: Established threat. Exploiting automated reporting thresholds to censor creators is a well-established tactic used by online trolls across various platforms.

Impact by dimension

• Physical security: Negligible. There was no threat to physical infrastructure, physical systems, or human safety during this incident.
• Information security: Minor. Trolls manipulated platform moderation algorithms to censor content, but the campaign lacked the scale, coordination, or state backing of national security information warfare.
• Sovereignty: Negligible. The incident had no impact on state authority, electoral processes, or government operations.
• Economic security: Negligible. Financial harm was limited to individual creator earnings, posing no threat to national economic stability or strategic technology.
• Societal stability: Minor. The incident demonstrates algorithmic bias and localized suppression of marginalized voices, but does not threaten broader national societal stability.