ByteDance Allegedly Trained "For You" Algorithm Using Content Scraped without Consent from Other Social Platforms

ByteDance allegedly scraped short-form videos, usernames, profile pictures, and descriptions of accounts on Instagram, Snapchat, and other sources, and uploaded them without consent on Flipagram, TikTok’s predecessor, in order to improve its “For You” algorithm's performance on American users.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

ByteDance created fake accounts using scraped public profiles to impersonate real users on Flipagram for corporate growth and advantage.

Additional risk subdomains

• 6.3 Economic and cultural devaluation of human effort: The creative video content of hundreds of thousands of users was duplicated without consent to build value for a competing platform.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by intentional decisions by ByteDance management and engineers to scrape competitor data as a growth hack for their active app.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The 'For You' algorithm is a personalization and recommendation system used in social media entertainment, which poses low or negligible risk under the EU AI Act.

AI system and alleged parties

• AI system: For You algorithm (ByteDance)
• AI purpose: Content Recommendation; Behavioral Modeling
• Behaviour type: unknown
• Alleged developer: ByteDance
• Alleged deployer: ByteDance
• Alleged harmed parties: Snapchat users, Instagram users, American social media users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Substantial, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Privacy

Reported: The report explicitly describes privacy concerns regarding the unauthorized scraping and duplication of user profiles and videos.

Directly caused: ByteDance scraped and duplicated usernames, profile pictures, and videos of hundreds of thousands of users without their knowledge or consent.

Psychological

Reported: The report explicitly describes psychological distress and concern from creators and parents regarding unauthorized impersonation.

Directly caused: Creators experienced distress and concern upon discovering that their likenesses and content were being impersonated on an unknown platform.

People affected

• Occurrences reported: 1
• People reportedly harmed: 200000
• People reportedly exposed: 200000

Potential causes

Management

• Growth Over Safety and Ethics: Management prioritized user growth and algorithm training over user privacy.
• Unethical OKR Targets: Set engineering OKRs to scrape 10,000 videos daily from priority countries.
• Deceptive Support Instructions: Instructed staff to hide scraping by blaming fans for unauthorized uploads.

Technology

• Algorithmic Training on Stolen Data: The For You algorithm was trained on unauthorized scraped US user data.
• Automated Scraping Tools: Engineering team used automated crawlers to scrape over 10,000 videos daily.

Data Inputs

• Unauthorized Data Scraping: Scraped videos and profiles from Instagram and Snapchat without consent.
• Use of Fake Accounts: Uploaded scraped data via fake accounts to mimic real user engagement.

Human Factors

• Creator Deception: Users were unaware their content was scraped and duplicated on Flipagram.
• Employee Compliance: Engineers followed growth hack OKRs despite terms of service violations.

Process and Methods

• Growth Hack Methodology: Prioritized rapid data acquisition and fake engagement metrics over ethics.
• Watermark and Resizing Edits: Resized videos and removed watermarks to hide the source of scraped data.

Regulatory Environment

• Weak Privacy Legislation: Lack of strong kids' privacy and safety laws allowed unchecked scraping.
• Inadequate Platform Defenses: Target platforms had weak technical barriers to prevent bulk data scraping.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly detailed and draw on multiple independent former employee sources and internal documents. ByteDance's official response is also included, providing a balanced view.
• Ambiguities identified: There is some ambiguity regarding whether the scraped data is still stored in China and the exact extent to which this data continues to influence the current TikTok algorithm.
• Alternative interpretations: The incident could be interpreted primarily as a corporate data theft and growth hack campaign rather than an AI safety failure, though the training of the recommendation algorithm connects it to AI.

In 2017, ByteDance allegedly scraped data from hundreds of thousands of US social media users to train its recommendation algorithm and populate its app Flipagram. While primarily a corporate privacy and terms of service violation, the incident highlights strategic concerns regarding foreign entities harvesting domestic user data to train influential personalization algorithms.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: ByteDance

Threat characteristics

• Imminence: Long-term. The incident occurred in 2017 and involves defunct platforms, representing a historical data-gathering event rather than an active, immediate crisis.
• Autonomy: Human-controlled. The data scraping and subsequent deployment of fake accounts were intentionally directed and executed by ByteDance employees and management.
• Novelty: Established threat. Data scraping and terms of service violations for platform growth are established industry practices, though applied here to train a proprietary algorithm.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic attacks, or critical infrastructure impacts were reported in connection with this data scraping incident.
• Information security: Minor. Involves unauthorized harvesting of US user data by a foreign-owned entity to train personalization algorithms, posing minor long-term strategic information control concerns.
• Sovereignty: Negligible. No direct compromise of government decision-making, electoral systems, or state authority was observed.
• Economic security: Minor. Represents unauthorized data acquisition from US tech companies to build competitive advantage for a foreign-owned algorithmic platform, violating terms of service.
• Societal stability: Minor. Involves mass privacy violations and unauthorized impersonation of hundreds of thousands of US social media users without their consent.