Underground Market for LLMs Powers Malware and Phishing Scams

A study by Indiana University researchers uncovered widespread misuse of large language models (LLMs) for cybercrime. Cybercriminals, according to that study, use LLMs like OpenAI's GPT-3.5 and GPT-4 to create malware, phishing scams, and scam websites. These models are available on underground markets, often bypassing safety checks through jailbreaking. Named malicious LLMs are BadGPT, XXXGPT, Evil-GPT, WormGPT, FraudGPT, BLACKHATGPT, EscapeGPT, DarkGPT, and WolfGPT.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.2 Cyberattacks, weapon development or use, and mass harm

The primary capability of the analyzed Mallas (93.4%) is the generation of sophisticated malware designed to evade detection, which constitutes using AI to develop cyber weapons.

Additional risk subdomains

• 4.3 Fraud, scams, and targeted manipulation: The Mallas are also heavily used to generate phishing emails and scam websites to defraud individuals and organizations.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The risk is caused by human cybercriminals intentionally jailbreaking and deploying LLMs post-deployment to facilitate cybercrime.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The systems are fundamentally general-purpose LLMs and chatbots, which fall under the transparency requirements of Limited Risk, though they are being jailbroken for malicious use.

AI system and alleged parties

• AI system: Claude-2, FraudGPT, GPT-3.5, GPT-4, WormGPT (Anthropic, Meta, OpenAI)
• AI purpose: Code Generation; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: XXXGPT, WormGPT, WolfGPT, FraudGPT, Evil-GPT, EscapeGPT, DarkGPT, Cybercriminals, BLACKHATGPT, BadGPT
• Alleged harmed parties: Organizations, internet users, Individuals targeted by malware

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the report explicitly describes the creation of malicious content such as phishing emails and scam websites.

Directly caused: The study found that WolfGPT and other Mallas generate highly convincing phishing emails and scam websites to deceive users.

Inferred additional harm: It is highly likely that millions of phishing emails and thousands of scam websites have been generated and distributed using these tools, exposing a vast number of internet users to malicious content.

Epistemic

Reported: Yes, the report mentions the exploitation of LLMs for creating false and misleading images and generating scam websites.

Directly caused: Mallas like EscapeGPT were used to generate highly convincing scam websites.

Inferred additional harm: The proliferation of AI-generated scam websites and phishing emails contributes to a broader erosion of trust in online communications and digital information.

People affected

• Occurrences reported: 1

Potential causes

Management

• Prioritizing Innovation over Security: Failure to balance rapid AI tool deployment with robust safety features.
• Inadequate API Safeguard Auditing: Insufficient testing of public APIs against known jailbreak techniques.
• Laissez-Faire Platform Management: Management fails to establish clear usage guidelines on hosting platforms.

Technology

• Model Jailbreaking Vulnerabilities: Susceptibility of public APIs like GPT-3.5 to jailbreak prompts.
• Uncensored Open-Source Models: Availability of models with minimal safety checks, like Pygmalion-13B.
• Sophisticated Malware Generation: Mallas generate compilable malware that evades VirusTotal detection.

Data Inputs

• NSFW Fine-Tuning Datasets: Use of unfiltered or NSFW datasets to fine-tune open-source models.
• Jailbreak Prompt Datasets: Exploitation of specific prompt pairs to bypass API safety filters.

Human Factors

• User Susceptibility to Phishing: Vigilant users can still be tricked by highly persuasive phishing emails.
• Developer Laissez-Faire Attitude: Hosting platforms providing unrestricted access without guidelines.
• Active Exploitation by Criminals: Cybercriminals actively using dark LLMs to automate phishing and fraud.

Process and Methods

• Inadequate Hosting Platform Rules: Platforms like FlowGPT lack clear usage guidelines and enforcement.
• Bypassing RLHF and Moderation: Mallas easily circumvent standard RLHF and API moderation mechanisms.
• Lack of Mandatory Incident Reporting: Organizations lack standardized processes for prompt cyber incident disclosure.

Regulatory Environment

• Absence of Stricter AI Regulations: Lack of mandated ethical development and robust security standards.
• Lack of Global Cyber Cooperation: Absence of unified international frameworks to track and prosecute criminals.
• Unrestricted Open-Source Access: Scientific-only models are publicly accessible without safety protocols.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide detailed, academic-backed evidence from a systematic study by Indiana University Bloomington, analyzing 212 Mallas and over 13,000 listings. The roles of the AI systems, the backend models used, and the specific capabilities (malware, phishing, scam sites) are explicitly documented.
• Ambiguities identified: The exact scale of successful cyberattacks executed using these specific Mallas (like FraudGPT or WormGPT) and the precise financial damages resulting from them are not quantified.
• Alternative interpretations: The Mallas could be viewed purely as traditional cybercrime tools that happen to use AI APIs, rather than distinct AI safety failures, though the ease of jailbreaking and availability of uncensored models points to systemic AI safety vulnerabilities.

The identification of a widespread underground ecosystem of malicious LLM services (Mallas) highlights how cybercriminals are leveraging AI to automate malware and phishing generation. While this lowers the technical barrier for cyberattacks and poses ongoing economic and security challenges, the threat is currently managed within standard cybersecurity frameworks and does not present an immediate crisis to critical national security functions.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Multiple nations
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The underground ecosystem represents an ongoing, strategic cyber capability concern rather than an active, time-critical national security crisis.
• Autonomy: Human-controlled. The AI models act as assistive tools for generating code and text, requiring human actors to prompt, deploy, and execute the cyberattacks.
• Novelty: Evolved capability. Represents a significant advancement in cybercrime automation by lowering technical barriers, though the underlying threat types (malware, phishing) are well-established.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic attacks, or critical infrastructure compromises were reported in connection with these malicious LLM services.
• Information security: Minor. The tools facilitate phishing and scam content generation, but there is no evidence of state-sponsored intelligence compromise or strategic information warfare.
• Sovereignty: Negligible. No disruption to core government operations, electoral systems, or state sovereignty was identified in the report.
• Economic security: Minor. While the services facilitate fraud and cybercrime like business email compromise, the impact remains within the scope of standard cyber defense and law enforcement procedures.
• Societal stability: Minor. The proliferation of phishing and identity theft exploits poses risks to individual privacy, but does not constitute systematic population-scale manipulation.