LLM Scrapers Allegedly Target Multiple Open Source Projects Disrupting the FOSS Ecosystem

In mid-March 2025, KDE's GitLab infrastructure was reportedly disrupted by aggressive AI web scrapers originating from Alibaba IP ranges. These bots allegedly ignored robots.txt and spoofed browser headers, which in turn purportedly overwhelmed the site and caused outages for developers. Similar incidents reportedly affected other FOSS projects like GNOME, SourceHut, and Fedora. The scraping is allegedly tied to large language model training, and reportedly imposes real costs and delays.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI crawlers lack the capability to respect standard web protocols (robots.txt) and rate-limits, while the LLMs lack the capability to generate accurate bug reports, leading to severe operational failures and resource exhaustion.

Additional risk subdomains

• 3.1 False or misleading information: AI systems generated hallucinated security vulnerability reports that appeared legitimate but were entirely false, wasting developer time.
• 6.3 Economic and cultural devaluation of human effort: AI companies aggressively extracted vast amounts of open-source data without consent or compensation, threatening the sustainability of the FOSS ecosystem.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The disruption and financial costs were caused by the automated actions of deployed AI web crawlers and LLM-generated bug reports, which were unintentional side effects of data harvesting and text generation goals.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The AI systems involved are web crawlers used for data harvesting and general-purpose language models, which do not fall under high-risk or unacceptable-risk categories of the EU AI Act.

AI system and alleged parties

• AI system: unspecified AI scraper bots
• AI purpose: Content Search; Writing Assistant
• Behaviour type: Autonomous
• Alleged developer: Unnamed generative AI companies, Alibaba
• Alleged deployer: Unnamed generative AI companies, Alibaba
• Alleged harmed parties: Sysadmins, SourceHut, Read the Docs, Linux Weekly News, KDE, Inkscape, GNOME, FOSS projects and communities, Fedora, Diaspora, Curl

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Psychological

Reported: The report explicitly describes psychological distress and frustration experienced by open-source maintainers.

Indirectly caused: Maintainers reported reaching a 'breaking point' and described their situation as a 'desperate cry for help', with sysadmins experiencing palpable desperation and stress due to persistent outages.

People affected

• Occurrences reported: 1
• People reportedly harmed: 15
• People reportedly exposed: 10000

Potential causes

Management

• Prioritizing data over sustainability: AI firms prioritize data gathering over externalized infrastructure costs.
• Unresponsive corporate behavior: Large AI firms remain unresponsive to complaints from project maintainers.
• Predatory resource extraction: Firms leverage massive capital while shifting bandwidth costs onto FOSS hosts.

Technology

• Evasion of standard blocking: Crawlers spoof user-agents and cycle through residential IP proxies.
• Expensive endpoint targeting: Crawlers target resource-heavy git blame and log pages repeatedly.
• AI-generated bug reports: LLMs generate realistic but hallucinated security vulnerability reports.

Data Inputs

• Ignoring robots.txt protocols: AI crawlers deliberately ignore robots.txt and spider configurations.
• High frequency data harvesting: Crawlers return to scrape the same pages as often as every six hours.
• Massive scale of scraper traffic: Bots generate terabytes of requests, representing up to 97% of site traffic.

Human Factors

• Wasted developer triage time: Maintainers spend valuable time refuting hallucinated security reports.
• Impact on legitimate human users: Defensive measures like proof-of-work challenges cause delays for real users.
• FOSS community resource constraints: Stretched-thin sysadmins must manually manage massive block lists.

Process and Methods

• Lack of rate-limiting by crawlers: AI firms fail to implement rate limits on their data harvesting runs.
• Public nature of FOSS infrastructure: Open collaboration model leaves git repositories publicly exposed to bots.
• Inadequate defensive tools: Traditional IP blocking fails against highly distributed botnets.

Regulatory Environment

• Absence of scraping regulations: No legal frameworks exist to enforce respectful crawling or data consent.
• Lack of enforcement of standards: No penalties for violating voluntary standards like robots.txt.
• Lack of consent or compensation: AI firms extract open-source data without clear consent or financial return.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide detailed, first-hand accounts from multiple open-source maintainers, complete with specific traffic statistics, financial impacts, and technical mitigation strategies. The dual facets of the incident (aggressive crawling and hallucinated bug reports) are well-documented across several independent projects.
• Ambiguities identified: The exact identity of some Chinese AI crawlers is not fully specified beyond their IP ranges (e.g., Alibaba ranges).
• Alternative interpretations: The incident could be viewed purely as a standard network security/DDoS issue rather than an AI safety issue, though the specific involvement of LLM training crawlers and LLM-generated bug reports makes it highly relevant to AI safety.

Aggressive data harvesting by AI developers has caused operational disruptions to key open-source software projects, threatening the sustainability of critical software supply chains. Additionally, LLM-generated hallucinated security reports waste developer resources, representing an evolved threat to strategic technological infrastructure.

• Overall national security impact: Substantial
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global open-source community
• Alleged perpetrator: AI technology corporations including OpenAI, Anthropic, Amazon, and Alibaba

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding the sustainability and security of open-source software supply chains rather than an immediate national security crisis.
• Autonomy: Full autonomy. AI web crawlers autonomously adapt to defensive measures by spoofing user agents, ignoring robots.txt protocols, and rotating IP addresses without human intervention.
• Novelty: Evolved capability. Represents a significant scaling and evolution of traditional web scraping and spamming techniques, now supercharged by the massive data demands of AI training.

Impact by dimension

• Physical security: Minor. Minor impact on digital infrastructure hosting. While open-source software is critical to global systems, the incident affected project repositories and development platforms rather than operational physical control systems.
• Information security: Minor. Flooding of FOSS projects with hallucinated AI-generated security reports represents a low-level information pollution threat, though it lacks evidence of being a coordinated state-sponsored intelligence operation.
• Sovereignty: Negligible. No direct impact on state sovereignty or core government functions, though some affected open-source software is utilized within government systems.
• Economic security: Substantial. Substantial threat to the sustainability of the open-source software ecosystem, which serves as a critical strategic technology foundation. Aggressive resource consumption, financial costs, and developer burnout threaten key software supply chains like Fedora, CPython, and Curl.
• Societal stability: Negligible. No significant impact on societal stability, civil liberties, or human rights.