Alleged Inclusion of 12,000 Live API Keys in LLM Training Data Reportedly Poses Security Risks

A dataset used to train large language models allegedly contained 12,000 live API keys and authentication credentials. Some of these were reportedly still active and allowed unauthorized access. Truffle Security found these secrets in a December 2024 Common Crawl archive, which spans 250 billion web pages. The affected credentials could have been exploited for unauthorized data access, service disruptions, financial fraud, and a variety of other malicious uses.

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.2 AI system security vulnerabilities and attacks

The report focuses on security vulnerabilities in AI systems, such as Wayback Copilot, jailbreaking, and prompt injection, which allow unauthorized access or manipulation.

Additional risk subdomains

• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: The vulnerabilities directly resulted in the exposure and potential leakage of private tokens, API keys, and secrets from thousands of organizations.
• 7.3 Lack of capability or robustness: The models fail to perform robustly against jailbreak attempts and can be manipulated into suggesting insecure code.

Causal factors

• Entity: Human
• Intent: Unintentional
• Timing: Post-deployment

The security risks stem from human actions, such as developers hard-coding credentials and training models on uncleaned datasets, leading to unintentional post-deployment vulnerabilities.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The systems described are general-purpose chatbots and AI assistants (such as Microsoft Copilot and ChatGPT) which pose moderate risk and are subject to transparency obligations under the Act.

AI system and alleged parties

• AI system: Microsoft Copilot, unspecified (Microsoft)
• AI purpose: Code Generation; Chatbot
• Behaviour type: Assistant
• Alleged developer: OpenAI, Microsoft, Common Crawl
• Alleged deployer: OpenAI, Microsoft Azure OpenAI Service, Microsoft, Common Crawl
• Alleged harmed parties: Tencent, Slack, PayPal, Microsoft, Mailchimp, Intel, IBM, Huawei, Google, AWS

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Privacy

Reported: The report explicitly describes privacy and credential exposure affecting thousands of organizations.

Directly caused: The Wayback Copilot vulnerability exposed over 300 private tokens, keys, and secrets belonging to various organizations.

Inferred additional harm: It is likely that additional private credentials and sensitive information of developers and organizations were exposed during the indexing of these repositories.

People affected

• Occurrences reported: 1
• People reportedly exposed: 16290

Potential causes

Technology

• Inability to Distinguish Secrets: LLMs cannot distinguish between valid and invalid secrets during training.
• CoT Reasoning Hijacking: Chain-of-thought intermediate reasoning can be hijacked to bypass safety.
• Improper Logit Bias Settings: Poorly adjusted logit biases allow uncensoring of restricted outputs.

Data Inputs

• Live Secrets in Training Data: Common Crawl datasets contain thousands of live authentication keys.
• Insecure Code in Fine-Tuning: Training LLMs on insecure code induces emergent misalignment.
• Cached Public Repositories: Bing indexes and caches repositories that were briefly public.

Human Factors

• Hard-Coded Credentials: Developers hard-coding credentials in code that gets scraped.
• Accidental Public Publishing: Users mistakenly publishing private repositories to public spaces.
• Multi-Turn Adversarial Inputs: Users exploiting multi-turn prompts to jailbreak LLM safety controls.

Process and Methods

• Inadequate Training Data Filtering: Lack of robust filtering to remove secrets and insecure code from datasets.
• Unsecured Search Engine Caching: Search engines caching sensitive data, exposing it to AI copilots.
• Flawed Safety Guardrails: Traditional safety guardrails fail to prevent emergent misalignment.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly outlines the technical vulnerabilities (Wayback Copilot, Common Crawl secrets) and references specific security firms (Truffle Security, Lasso Security, Palo Alto Networks) and academic research on emergent misalignment. The details of the vulnerabilities are explicit and well-documented.
• Ambiguities identified: None significant, though the real-world exploitation of these secrets by malicious actors is not detailed.
• Alternative interpretations: The incident could be viewed purely as a data-leakage issue rather than an AI safety issue, but the role of LLMs in retrieving and suggesting these secrets makes it an AI safety concern.

Widespread security vulnerabilities in AI systems, including the exposure of 12,000 live API keys in training data and the 'Wayback Copilot' exploit affecting over 16,000 organizations, present substantial risks to economic and technological security. While no active state-sponsored exploitation is detailed, the systemic nature of these vulnerabilities highlights significant supply-chain risks in AI deployment.

• Overall national security impact: Substantial
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Multiple nations
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Near-term. The exposure of live credentials and vulnerable AI systems requires near-term remediation, patching, and credential rotation to prevent active exploitation.
• Autonomy: Human-controlled. The affected AI systems function as assistants under direct human control, though they retrieve and generate data based on user prompts.
• Novelty: Evolved capability. Represents an evolved threat landscape where traditional credential leaks are compounded by AI training data ingestion and LLM-based retrieval techniques.

Impact by dimension

• Physical security: Negligible. No kinetic attacks, critical infrastructure manipulation, or physical safety threats were reported as a result of these vulnerabilities.
• Information security: Substantial. Exposure of over 300 private tokens and 12,000 live API keys, including AWS root keys, from thousands of organizations poses a substantial risk to data security and intelligence integrity.
• Sovereignty: Negligible. The incident involves commercial AI systems and developer repositories, with no reported direct impact on government decision-making, elections, or state sovereignty.
• Economic security: Substantial. Exposure of private repositories and tokens belonging to major global technology companies like Google, Microsoft, and IBM threatens intellectual property and technological security.
• Societal stability: Negligible. No mass surveillance, systematic discrimination, or civil unrest was reported; the impact is primarily technical and corporate.