Grant Reviewers Fed Applications into Generative AI to Produce Reports, Allegedly Breaching Confidentiality

Peer reviewers of Australian government grant applications inserted applicants' work into generative AI systems such as ChatGPT to generate assessment reports, which allegedly posed confidentiality and security issues.

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information

The incident involved the unauthorized uploading of confidential, unpublished grant proposals containing sensitive research ideas to an external AI system, violating confidentiality agreements.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Reviewers inappropriately relied on ChatGPT to generate academic assessments, substituting automated text generation for rigorous human peer review.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The confidentiality breach and poor-quality reviews were caused by human peer reviewers intentionally choosing to upload sensitive proposals to ChatGPT post-deployment.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: Limited Risk. ChatGPT is a generative AI chatbot, which falls under the limited risk category requiring transparency obligations so that users are aware they are interacting with AI-generated content.

AI system and alleged parties

• AI system: ChatGPT (OpenAI)
• AI purpose: Writing Assistant; Question Answering
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI
• Alleged harmed parties: research grant applicants, research grant administrators

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Epistemic

Reported: Yes, the reports describe peer reviews that were generic regurgitations of proposals without actual critique or insight.

Directly caused: ChatGPT generated bland, uncritical summaries instead of rigorous peer reviews for at least 4 proposals, degrading the quality of the evaluation.

Indirectly caused: This practice undermines the integrity of the academic peer review process and scientific funding decisions.

Inferred additional harm: Other reviews may have contained hallucinated criticisms or praise, further corrupting the funding decision process.

People affected

• Occurrences reported: 8
• People reportedly harmed: 8
• People reportedly exposed: 8

Potential causes

Management

• Unmanaged Academic Workloads: Universities failed to allocate formal time for peer review activities.
• Slow Policy Updates: Management delayed issuing clear guidelines on generative AI usage.
• Failure to Pre-empt Risks: ARC management did not anticipate AI risks despite community warnings.

Technology

• Availability of ChatGPT: Easy access to generative AI tools allowed rapid text generation.
• Lack of AI Detection Tools: No automated systems were used to identify AI-generated review text.

Data Inputs

• Confidential Data Uploads: Reviewers input confidential grant proposal text into external AI tools.

Human Factors

• Academic Overwork: High pressure and lack of time led reviewers to cut corners using AI.
• Reviewer Academic Misconduct: Dishonorable behavior where reviewers relied on AI rather than expert analysis.

Process and Methods

• Excessive Proposal Length: Proposals of 100-150 pages created massive reading workloads for reviewers.
• Inadequate Quality Control: No internal checks to filter out generic or AI-generated feedback.
• Lack of Score Transparency: Inability for applicants to see scores made detecting AI reviews harder.

Regulatory Environment

• Absence of Explicit AI Rules: Existing policies did not explicitly mention or ban generative AI tools.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are clear, from reputable sources, and contain direct quotes and evidence of the incident, such as the 'Regenerate response' button text appearing in the review.
• Ambiguities identified: The exact number of affected proposals is not fully known, only that ARC Tracker verified 4 and suspected 4 more.

Australian Research Council peer reviewers compromised the confidentiality of sensitive, unpublished grant proposals by uploading them to ChatGPT. While this represents a minor breach of government information security and compromises the integrity of the public funding process, the national security impact remains low and manageable through updated policy enforcement.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: Australia
• Alleged perpetrator: Academic peer reviewers

Threat characteristics

• Imminence: Long-term. The incident represents a policy compliance and data governance concern rather than an active, imminent national security crisis.
• Autonomy: Human-controlled. The AI tool was intentionally used by human reviewers to assist with their work; the system did not act autonomously.
• Novelty: Evolved capability. Represents an evolved data leakage threat where users upload proprietary government and academic data to external generative AI platforms.

Impact by dimension

• Physical security: Negligible. No physical threats, kinetic targeting, or critical infrastructure impacts were reported in connection with this incident.
• Information security: Minor. Unpublished, highly sensitive research proposals were uploaded to an external commercial AI platform, representing a minor breach of information security and confidentiality.
• Sovereignty: Minor. The integrity of a government funding agency's decision-making process was compromised by reviewers using unauthorized AI tools to generate assessments.
• Economic security: Minor. Potential exposure of proprietary scientific research and technological concepts to an external commercial entity, posing minor risks to domestic technological advantage.
• Societal stability: Negligible. No significant threats to social cohesion, civil liberties, or population safety were identified.