ChatGPT-Written Bug Reports Deemed "Nonsense" by White Hat Platform, Prompted Bans

ChatGPT-generated responses submitted to smart contract bug bounty platform Immunefi reportedly lacked details to help diagnose technical issues, which reportedly wasted the platform's time, prompting bans to submitters.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The 15 banned users used ChatGPT to generate fake bug reports in an attempt to cheat the Immunefi bug bounty system for financial gain.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI system generated highly polished but technically nonsensical reports, demonstrating a lack of capability to perform the task accurately.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by human users intentionally using ChatGPT to generate and submit fake bug reports to a bounty platform for personal gain.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: ChatGPT is a chatbot and generative AI tool, which falls under the transparency obligations of Limited Risk systems under the EU AI Act.

AI system and alleged parties

• AI system: ChatGPT 3 (OpenAI)
• AI purpose: Writing Assistant; Technical Text Generation
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI, Immunefi users
• Alleged harmed parties: Immunefi

Harm severity

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly exposed: 15

Potential causes

Management

• Lack of Clear AI Usage Guidelines: Platform had to reactively ban users due to lack of upfront AI policies.

Technology

• Plausible but Inaccurate Output: Polished presentation successfully disguised technically incorrect answers.
• Lack of Technical Reasoning: The tool has no real capability to identify software bugs.

Data Inputs

• Biased Training Data: Biased training data causes answers to lack common sense and context.

Human Factors

• Overreliance on AI Tools: Users submitted AI-generated reports without verifying their accuracy.
• Financial Incentive Exploitation: Users tried to exploit the bounty system by spamming automated reports.

Process and Methods

• Absence of Pre-submission Validation: No manual review of AI output was done by the submitters before reporting.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, consistent details about the incident, including the platform involved, the number of banned users, the reason for the ban, and the specific limitations of the AI tool.

Immunefi banned 15 users for submitting ChatGPT-generated bug reports that were technically nonsensical but polished. This incident represents low-level AI-enabled spam wasting developer resources on a private platform, posing negligible threats to national security.

• Overall national security impact: Negligible
• Response level: Minor
• Scope: Unknown
• Primary target: No clear primary
• Alleged perpetrator: 15 banned individuals

Threat characteristics

• Imminence: Long-term. The issue of AI-generated spam represents an ongoing, low-level operational challenge rather than an imminent national security crisis.
• Autonomy: Human-controlled. The AI tool was directly prompted and utilized by human actors to generate the spam reports.
• Novelty: Evolved capability. Represents an evolution of spamming techniques where LLMs are used to generate superficially plausible technical text.

Impact by dimension

• Physical security: Negligible. The incident involved low-quality bug reports submitted to a private decentralized finance platform, with no threat to physical systems or critical infrastructure.
• Information security: Negligible. No foreign intelligence operations, systematic disinformation campaigns, or classified data compromises were involved.
• Sovereignty: Negligible. The incident was limited to a private bug bounty platform and did not affect any government systems, elections, or sovereignty.
• Economic security: Negligible. While developer time was wasted reviewing nonsensical reports, there was no strategic technological theft or significant threat to national economic stability.
• Societal stability: Negligible. No human rights violations, mass surveillance, or threats to societal stability occurred.