Meta AI's Scientific Paper Generator Reportedly Produced Inaccurate and Harmful Content

Meta AI trained and hosted a scientific paper generator that sometimes produced bad science and prohibited queries on topics and groups that are likely to produce offensive or harmful content.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The primary issue was Galactica generating highly authoritative-sounding but completely false scientific information, fake papers, and incorrect facts, which could mislead users.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The model's safety filters censored queries about AIDS and queer theory, which critics argued further marginalized and misrepresented minority communities.
• 7.3 Lack of capability or robustness: The model exhibited a fundamental lack of capability to distinguish truth from falsehood, leading to hallucinations and incorrect scientific outputs.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The generation of false scientific facts and biased content was an unexpected outcome of the AI system's operation after it was deployed as a public demo.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a generative AI chatbot. Under the EU AI Act, chatbots and generative models are classified as Limited Risk, requiring transparency to inform users they are interacting with AI.

AI system and alleged parties

• AI system: Galactica (Meta)
• AI purpose: Writing Assistant; Technical Text Generation
• Behaviour type: Assistant
• Alleged developer: Meta AI, Meta, Facebook
• Alleged deployer: Meta AI, Meta, Facebook
• Alleged harmed parties: Minority Groups, Meta AI, Meta, Facebook

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes that users could generate racist or potentially offensive prompts, producing authoritative-sounding toxic content.

Directly caused: The model generated biased, offensive, and toxic content when prompted by users during the three days the demo was active.

Inferred additional harm: Some users on social media may have been exposed to offensive or racist outputs shared by others, though the exact number is unquantified.

Differential treatment

Reported: The report explicitly describes how the model's safety filters censored content about minority communities, such as queer theory and AIDS.

Directly caused: The model's non-transparent safety filters blocked queries about AIDS and queer theory, treating these legitimate scientific topics as unsafe.

Inferred additional harm: This censorship could lead to the exclusion and further marginalization of minority groups in scientific AI tools.

Epistemic

Reported: The report explicitly describes the model generating realistic-sounding nonsense, incorrect scientific facts, and fabricated citations.

Directly caused: The model generated false scientific papers, incorrect dates, and fake citations that sounded highly authoritative, misleading users who tested it.

Inferred additional harm: If the demo had remained online, it could have severely polluted the scientific information ecosystem and eroded trust in peer-reviewed research.

People affected

• Occurrences reported: 1
• People reportedly exposed: 1000

Potential causes

Management

• Hasty Public Deployment: Released a public demo without thoroughly assessing safety risks.
• Unclear Use Case Definition: Failed to clearly document and restrict intended use cases.

Technology

• Stochastic Parrot Limitations: Model captures word patterns but cannot tell fact from fiction.
• Flawed Content Filtering: Filters censored benign minority topics while letting toxic prompts pass.
• Lack of Output Provenance: The UI lacked features to flag or identify model-generated content.

Data Inputs

• Statistical Pattern Training: Training on 48M papers led to mimicry rather than semantic understanding.
• Inherent Training Data Biases: The training data led to the reproduction of prejudice and stereotypes.

Human Factors

• Adversarial Prompting by Users: Users entered racist or offensive prompts to test model limits.
• Blind Trust in Model Outputs: Users may treat outputs as authoritative without verifying facts.
• Defensive Developer Attitude: Management blamed users for 'casually misusing' the demo.

Process and Methods

• Misleading Product Hype: Marketing framed a research model as a solved scientific tool.
• Inadequate UI Safeguards: The UI styled outputs like real papers without clear disclaimers.
• Poor Issue Reporting Channels: No structured way for community feedback other than public Twitter.

Regulatory Environment

• Lack of Generative AI Standards: No clear regulatory framework governs the release of LLM demos.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the model's release, its specific failures (hallucinations, censorship, bias), and its subsequent takedown. There is high agreement among the sources.
• Ambiguities identified: None of significance.
• Alternative interpretations: The incident could be interpreted primarily as a lack of capability (7.3) rather than misinformation (3.1), but the core risk realized was the generation of false information.

Meta AI's short-lived public release of the Galactica LLM highlighted risks of automated scientific misinformation and biased outputs. While the incident caused epistemic concern within the scientific community and minor societal concerns due to biased filtering, it posed negligible threats to core national security areas like critical infrastructure, sovereignty, or intelligence.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The public demo was taken offline within three days, neutralizing immediate risks. Any remaining concerns are strategic and long-term regarding LLM reliability.
• Autonomy: Human-controlled. The AI functioned as a writing assistant, responding directly to human prompts and requiring human oversight to utilize or verify its outputs.
• Novelty: Evolved capability. The incident demonstrated an evolved capability of LLMs trained on massive scientific datasets to generate highly authoritative-sounding but completely fabricated technical information.

Impact by dimension

• Physical security: Negligible. The incident involved a software-based language model demo with no connection to physical systems, critical infrastructure, or kinetic capabilities.
• Information security: Negligible. While the model generated false scientific papers, there is no evidence of coordinated information warfare, intelligence compromise, or state-sponsored disinformation campaigns.
• Sovereignty: Negligible. No impact on state authority, government decision-making, electoral systems, or constitutional processes was observed.
• Economic security: Negligible. The incident represents a public relations and product failure for a private company, with no significant threat to national economic stability or strategic industries.
• Societal stability: Minor. The model's opaque safety filters censored legitimate topics like AIDS and queer theory, and generated biased content, presenting minor risks of marginalization and societal bias.