OpenAI's 4o Model Allegedly Used to Generate Fake Receipts and Prescriptions

OpenAI's new image generator, 4o, was reportedly used to produce realistic fraudulent documents, including fake restaurant receipts, prescriptions for controlled substances, and potentially other identity or financial documents. Users demonstrated how the model could be prompted to bypass safety restrictions to be misused for expense fraud, medical forgery, and other various forms of document falsification.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The incident involves using GPT-4o to generate highly realistic fake receipts and medical prescriptions, which directly facilitates financial fraud, expense scams, and the illicit acquisition of controlled substances.

Additional risk subdomains

• 3.1 False or misleading information: The AI system generates highly realistic but entirely fabricated documents containing false information.

Causal factors

• Entity: AI
• Intent: Intentional
• Timing: Post-deployment

The risk arises post-deployment from the AI system's advanced capability to generate highly realistic text within images, which is intentionally leveraged by users to create fraudulent documents.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3. Limited Risk: AI systems that pose a moderate risk and require specific transparency obligations. The report describes GPT-4o as an image-generating model capable of creating realistic fake documents, which falls under AI-generated content.

AI system and alleged parties

• AI system: GPT-4o 4o (OpenAI)
• AI purpose: Image Generation; Deepfake Image Generation
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: Unknown scammers
• Alleged harmed parties: Regulatory agencies, Pharmacists, Law enforcement, healthcare providers, General public, Expense management platforms, Employers

Harm severity

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the report describes the creation of malicious content in the form of fraudulent documents.

Directly caused: The AI system directly generated a fake receipt for a San Francisco steakhouse and fake prescriptions for controlled substances like Zoloft.

Indirectly caused: N/A

Inferred additional harm: It is highly likely that other users have generated similar fraudulent documents, potentially exposing numerous organizations to expense fraud, though specific numbers are not available.

Epistemic

Reported: Yes, the report describes the generation of highly realistic fake evidence and documents that undermine visual trust.

Directly caused: The AI generated a fake receipt with accurate calculations and fake medical prescriptions, eroding the reliability of images as proof of real-world events.

Indirectly caused: N/A

Inferred additional harm: The widespread availability of this capability is likely to cause systemic epistemic harm, making it extremely difficult for organizations to verify the authenticity of any digital document or image.

People affected

• Occurrences reported: 1
• People reportedly exposed: 2

Potential causes

Technology

• Advanced Text-in-Image Generation: GPT-4o generates realistic text within images, enabling convincing forgery.
• Easily Bypassed Guardrails: Metadata tags and watermarks have simple workarounds.

Human Factors

• High Propensity for Fraud: Historical data shows a high rate of individuals willing to falsify receipts.
• Overreliance on Visual Trust: Organizations rely on images of documents as absolute proof of validity.

Process and Methods

• Weak Internal Controls: Flawed verification processes fail to detect fake documents.

Information quality

• Classification confidence: High
• Reason for confidence: The reports clearly describe the capabilities demonstrated by the users and the specific documents generated (receipts, prescriptions). There is no ambiguity about the AI model involved (GPT-4o) or the nature of the risk (document fraud). While the reports discuss potential future harms rather than actual realized financial losses, this distinction is clear and easily accounted for in the assessment.
• Ambiguities identified: None. The reports are consistent and clearly describe a capability demonstration rather than a real-world exploit with actual victims.
• Alternative interpretations: The incident could be interpreted as a security vulnerability demonstration rather than a fraud incident, but the primary focus is on the generation of fraudulent content.

Researchers demonstrated that OpenAI's GPT-4o can generate highly realistic fraudulent documents like receipts and prescriptions. While this poses a minor threat to financial, administrative, and medical verification processes, it currently represents a capability demonstration rather than an active national security crisis.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Multiple nations
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Near-term. The model is currently deployed and active, presenting an ongoing capability that could be exploited in the near term.
• Autonomy: Human-controlled. The AI acts as an assistant, generating images based on direct human prompts and requiring human execution to commit fraud.
• Novelty: Evolved capability. Represents an evolved capability in AI image generation, specifically regarding the high-fidelity rendering of coherent text within images.

Impact by dimension

• Physical security: Negligible. Minimal direct threat to physical systems or critical infrastructure, though potential illicit acquisition of controlled substances via fake prescriptions is noted.
• Information security: Minor. Enables sophisticated document forgery which could be used in intelligence or disinformation contexts, but no active state-sponsored campaigns are reported.
• Sovereignty: Minor. Potential for generating fake IDs, tax forms, and birth certificates poses a minor risk to administrative government functions and verification processes.
• Economic security: Minor. Poses a minor threat to financial systems and corporate expense verification through realistic fake receipts, checks, and financial documents.
• Societal stability: Minor. Erodes trust in digital documents and visual evidence, contributing to epistemic challenges, but has not caused widespread societal disruption.