Preprints Reportedly from Researchers from Multiple Universities Allegedly Contain Covert AI Prompts

Hidden prompts reportedly were discovered in at least 17 academic preprints on arXiv that purportedly instructed AI tools to deliver only positive peer reviews. The lead authors are reportedly affiliated with 14 institutions in eight countries, including Waseda University, KAIST, Peking University, and the University of Washington. The alleged concealed instructions, some of which were reportedly embedded using white text or tiny fonts, were purportedly intended to influence any reviewers who rely on AI tools.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The researchers used hidden prompt injections to manipulate AI-based evaluation tools to gain an unfair personal advantage (positive peer reviews) over others, which constitutes cheating.

Additional risk subdomains

• 2.2 AI system security vulnerabilities and attacks: The incident relies on exploiting indirect prompt injection vulnerabilities in AI systems to manipulate their outputs.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The risk was caused by human researchers intentionally embedding hidden prompts in their papers to manipulate AI systems post-deployment.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: Limited Risk. The AI systems being manipulated are general-purpose language models and chatbots used for reading, summarizing, and reviewing documents, which fall under limited risk transparency obligations.

AI system and alleged parties

• AI system: AI tools
• AI purpose: Content Verification; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: Unnamed large language model developers
• Alleged deployer: Unnamed peer reviewers, Unnamed conference paper reviewers
• Alleged harmed parties: Research community, Peer review process, Academic integrity, Academic conferences

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Substantial, indirect Substantial
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Epistemic

Reported: The report explicitly describes epistemic harm, noting that hidden prompts can cause AI tools to output incorrect summaries and keep users from accessing the right information.

Directly caused: The hidden prompts directly manipulated AI tools into generating biased, exclusively positive reviews of academic papers, distorting the objective evaluation process.

Indirectly caused: This manipulation could lead to the publication of unverified or low-quality research, undermining the integrity of the scientific literature and public trust in academic peer review.

Inferred additional harm: It is highly likely that similar hidden prompt injection techniques are being used on other web documents, leading to widespread, undetected distortion of information retrieved by AI search assistants.

People affected

• Occurrences reported: 17
• People reportedly exposed: 51

Potential causes

Management

• Lack of Institutional Guidelines: Universities failed to establish rules for appropriate AI usage.
• Unmanaged Reviewer Workload: Rising manuscript volume pressures reviewers to outsource to AI.

Technology

• Indirect Prompt Injection Vulnerability: AI models read and execute hidden text instructions within documents.
• Lack of Technical Defenses: AI providers lack technical measures to filter hidden text prompts.

Data Inputs

• Concealed Text in Manuscripts: White text or tiny fonts hide instructions from humans but not AI.

Human Factors

• Reviewer Reliance on AI: Overburdened peer reviewers use AI tools despite explicit bans.
• Deliberate Manipulation by Authors: Authors insert prompts to force positive evaluations or test AI.

Process and Methods

• Inconsistent AI Policies: No unified rules exist among academic conferences and journals.
• Absence of Automated Checks: Submission systems fail to scan for white text or hidden prompts.

Regulatory Environment

• Lack of Unified AI Standards: No industry-wide regulations govern ethical AI usage in academia.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, factual details about the discovery of hidden prompts in 17 academic papers across 14 institutions. The mechanism of the exploit (white text, small fonts) and the intent (manipulating AI reviewers) are explicitly documented, with statements from involved researchers and institutions.
• Ambiguities identified: The specific AI models used by the reviewers were not identified in the report.
• Alternative interpretations: Some authors argue the prompts were a protest/check against 'lazy reviewers' using AI, rather than an attempt to cheat, though the operational effect remains the manipulation of the evaluation process.

Researchers from multiple global universities embedded hidden white-text prompts in academic preprints to manipulate AI peer-review tools. While primarily an academic integrity issue, it highlights the real-world exploitability of indirect prompt injection vulnerabilities in document processing systems, which has minor implications for broader information security.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: United States, China, Japan, South Korea, Singapore
• Alleged perpetrator: Academic researchers

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding AI vulnerabilities and academic integrity rather than an active crisis.
• Autonomy: Human-supervised. AI tools acted as assistants to human reviewers who ultimately remained in the loop to make final publication decisions.
• Novelty: Evolved capability. Indirect prompt injection is a known vulnerability, but applying it via hidden white text in academic preprints to manipulate peer review represents an evolved application.

Impact by dimension

• Physical security: Negligible. No physical systems, infrastructure, or human safety elements were affected or targeted in this academic incident.
• Information security: Minor. Demonstrates indirect prompt injection vulnerability that can manipulate AI document summarizers. While limited to academic peer review here, the technique could be used to distort information retrieval in broader intelligence contexts.
• Sovereignty: Negligible. No core government operations, electoral systems, or sovereign state authority were impacted by this academic integrity incident.
• Economic security: Minor. Targeted prestigious research institutions globally. While it threatens the integrity of scientific literature and AI research evaluation, the immediate economic and technological security impact remains minor.
• Societal stability: Negligible. No threats to civil liberties, mass surveillance, or social cohesion were reported. Impact is confined to the academic publishing community.