GPT-2 Able to Recite PII in Training Data

OpenAI's GPT-2 reportedly memorized and could regurgitate verbatim instances of training data, including personally identifiable information such as names, emails, twitter handles, and phone numbers.

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information

The incident involves language models memorizing and leaking personally identifiable information (PII) such as contact details and private chat logs from their training data.

Additional risk subdomains

• 6.3 Economic and cultural devaluation of human effort: The models memorized and reproduced verbatim copyrighted text from books and software source code, raising copyright infringement concerns.
• 3.1 False or misleading information: The model generated a false news story incorrectly attributing a real murder to an unrelated murder victim.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk of data leakage is caused by the AI system's unintentional regurgitation of training data during post-deployment querying.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes GPT-2 and GPT-3, which are general-purpose generative AI models and chatbots. Under the EU AI Act, chatbots and generative AI systems that interact with humans or generate synthetic content are classified as Limited Risk, requiring specific transparency obligations to notify users that they are interacting with AI.

AI system and alleged parties

• AI system: GPT-2, GPT-3 (OpenAI)
• AI purpose: Writing Assistant; Code Generation
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI
• Alleged harmed parties: people having personal data in GPT-2's training data, OpenAI

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic or malicious content created or spread directly by the incident.

Directly caused: GPT-2 generated fictitious IRC conversations about transgender rights using real usernames from leaked GamerGate logs.

Privacy

Reported: The report explicitly describes privacy violations caused directly by the incident.

Directly caused: GPT-2 and GPT-3 regurgitated verbatim PII, including the contact information of a specific individual (work address, email, phone, fax) and a real community center's phone number, as well as leaked private GamerGate IRC logs.

Inferred additional harm: Given that 0.1% of generations contain verbatim training data and 13% of memorized examples contain names or contact info, it is highly likely that many other individuals' private data was exposed to users querying these models.

Epistemic

Reported: The report explicitly describes epistemic harm caused directly by the incident.

Directly caused: GPT-2 generated a false news story incorrectly attributing the murder of M. R. to A. D., who was actually a victim in an unrelated crime.

People affected

• Occurrences reported: 1
• People reportedly harmed: 4
• People reportedly exposed: 78

Potential causes

Management

• Commercialization Pressure: Prioritizing rapid API commercialization over comprehensive safety.
• Underfunded Data Collection: Investing heavily in model scaling while underfunding data curation.
• Inadequate Risk Assessment: Underestimating the feasibility of training data extraction attacks.

Technology

• Model Memorization Capacity: Large language models memorize and regurgitate rare training data.
• Lack of Output Filtering: Initial models lacked systems to block or rewrite sensitive outputs.
• Differential Privacy Limits: Standard DP is ineffective for data repeated across multiple sites.

Data Inputs

• Uncurated Web Scraping: Scraped training datasets contain leaked PII and copyrighted text.
• Lack of Data Sanitization: No effective automated process to clean sensitive data from corpus.
• Inclusion of Copyrighted Works: Books and source code were included in the training dataset.

Human Factors

• Adversarial Prompting: Users can craft specific queries to extract memorized training data.
• Lack of Output Curation: Deploying raw model outputs directly to users without human review.
• Context Misalignment: Using web-scraped data outside of its originally intended context.

Process and Methods

• Flawed Dataset Curation: Relying on broad web scrapes instead of carefully curated datasets.
• Inadequate Testing for Leakage: Lack of rigorous testing to identify memorized data before release.
• Lack of Robust Post-Processing: Absence of real-time filters to detect and replace sensitive data.

Regulatory Environment

• Unclear Data Deletion Laws: Legal gray area regarding right to be forgotten in trained weights.
• Ambiguous Fair Use Standards: Lack of clear legal boundaries for training models on copyrighted data.
• Inadequate Privacy Protections: Existing frameworks like GDPR do not easily map to neural networks.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly detailed, based on peer-reviewed academic research ('Extracting Training Data from Large Language Models') and technical journalism. They provide specific statistics, methodology, and concrete examples of the memorized data (PII, copyrighted text, and false attributions). There is no conflicting information regarding the core findings.
• Ambiguities identified: The exact total number of individuals whose PII was memorized across the entire model is not fully quantified, only estimated via sampling.
• Alternative interpretations: None. The technical mechanism of memorization and regurgitation is well-established and agreed upon by both the researchers and OpenAI.

Researchers demonstrated that GPT-2 and GPT-3 can memorize and regurgitate sensitive training data, including personally identifiable information and copyrighted content. This vulnerability poses minor national security risks, primarily concerning individual privacy and intellectual property protection, rather than active threats to critical infrastructure or state sovereignty.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Data memorization is an ongoing, structural characteristic of large language models requiring long-term technical and regulatory solutions.
• Autonomy: Human-controlled. The AI operates as a writing assistant, generating text only in response to direct human prompts and queries.
• Novelty: First-of-its-kind. This was a pioneering academic demonstration showing that large language models systematically memorize and can be forced to regurgitate sensitive training data.

Impact by dimension

• Physical security: Negligible. No threats to physical systems, infrastructure, or human safety were reported or indicated in this incident.
• Information security: Minor. The incident involves leakage of publicly scraped personal data and copyrighted material, posing privacy risks but no compromise of classified intelligence or active state-sponsored information warfare.
• Sovereignty: Negligible. There is no indication of threats to state authority, electoral systems, or core government decision-making processes.
• Economic security: Negligible. While copyright issues and intellectual property regurgitation occurred, they do not threaten strategic national industries or overall economic security.
• Societal stability: Minor. The models exposed personally identifiable information and leaked private logs, representing individual privacy violations, but did not cause large-scale societal instability or systemic civil rights abuses.