Scammers Reportedly Using Deepfakes of Health Experts and Public Figures in Australia to Sell Health Supplements and Give Harmful Advice

Scammers are reportedly harnessing AI-generated deepfakes of health experts and public figures in Australia in order to sell health supplements and give harmful health advice. Among the reported cases, deepfake videos are alleged to have falsely depicted Jonathan Shaw and Karl Stefanovic endorsing "Glyco Balance" for diabetes management, while Karl Kruszelnicki reportedly was falsely shown promoting blood pressure pills.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The incident involves malicious actors using generative AI to impersonate trusted medical professionals and public figures to scam individuals into purchasing unapproved supplements.

Additional risk subdomains

• 3.1 False or misleading information: The deepfakes spread dangerous medical misinformation, falsely claiming standard diabetes treatments are dangerous and promoting unproven supplements.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by human scammers who intentionally deployed generative AI tools post-deployment to create deceptive deepfakes for financial gain.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes the use of AI-generated deepfakes to impersonate medical professionals, which falls under the transparency obligations of Limited Risk systems.

AI system and alleged parties

• AI system: ChatGPT, unspecified (OpenAI)
• AI purpose: Deepfake Video Generation; Voice Generation
• Behaviour type: Tool
• Alleged developer: Unknown deepfake technology developers
• Alleged deployer: Unknown scammers
• Alleged harmed parties: Karl Stefanovic, Karl Kruszelnicki, Jonathan Shaw, General public, Diabetes patients

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes malicious deepfake videos and fake articles created to deceive the public.

Directly caused: Scammers created highly realistic deepfake videos of Professor Shaw and Karl Stefanovic, as well as fake interviews and articles on ResearchGate.

Inferred additional harm: Scammers likely generated hundreds of variations of these deepfakes and fake reviews across multiple platforms, exposing thousands of social media users to malicious content.

Psychological

Reported: The report describes distress and concern experienced by the targeted professional and a victim.

Indirectly caused: Professor Shaw experienced distress and reputational anxiety due to his likeness being used to promote dangerous medical advice. Michael experienced regret and distress after realizing he was scammed.

Inferred additional harm: Dozens of other targeted individuals likely experienced distress, anxiety, and regret upon realizing they were defrauded or misled about their health.

Epistemic

Reported: The report explicitly describes the spread of medical misinformation and the erosion of trust in legitimate health professionals.

Directly caused: The deepfakes spread false claims that metformin is dangerous and that Glyco Balance stabilizes blood sugar in 24 hours, undermining public trust in medical advice.

Indirectly caused: Professor Shaw noted that this makes it much more difficult for ordinary members of the public to identify trusted sources, eroding the shared information ecosystem.

Inferred additional harm: Widespread dissemination of doctor deepfakes likely causes systemic erosion of trust in public health institutions and medical professionals across Australia.

People affected

• Occurrences reported: 1
• People reportedly harmed: 3
• People reportedly exposed: 1000

Potential causes

Management

• Safety Resource Underinvestment: Platforms prioritized rapid growth over investing in robust security tools.
• Reactive Abuse Reporting: Management relied on slow user reporting rather than proactive prevention.
• Inadequate Doctor Protections: Platform safety tools focused on celebrities rather than medical experts.

Technology

• Generative AI Deepfakes: Generative AI allowed scammers to create highly realistic fake videos easily.
• AI Voice Cloning: Scammers accurately cloned the voices of trusted medical professionals.
• Limited Facial Recognition: Meta protective facial recognition is limited to select public figures.

Data Inputs

• Stolen Video and Audio Data: Publicly available media was harvested to train AI deepfake models.
• Fake Research Publications: Scammers uploaded fake reviews to platforms like ResearchGate to build trust.
• Misleading Online Reviews: AI-generated testimonials were indexed online to create false credibility.

Human Factors

• High Trust in Medical Experts: Patients naturally trust advice from well-known medical professionals.
• Cognitive Fatigue: Users struggle to maintain constant vigilance when evaluating online ads.
• Vulnerable Demographics: Elderly patients with worsening symptoms are highly susceptible to scams.

Process and Methods

• Slow Moderation Response: Meta took nine days and required intellectual property claims to remove ads.
• Whack-a-Mole Proliferation: Scammers rapidly post new fraudulent ads as soon as old ones are deleted.
• Ineffective Auto-Detection: Automated systems failed to catch medical impersonation scams proactively.

Regulatory Environment

• Jurisdictional Gaps: Regulators struggle to police international cybercriminals hosting scams.
• Inadequate Platform Liability: Lack of strict legal penalties for hosting fraudulent AI-generated ads.
• Lax Identity Verification: Platforms do not mandate strict identity checks for publishing content.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, detailed, and verified accounts of the deepfake scam, including a specific victim (Michael) and the targeted professional (Professor Shaw). The role of generative AI in creating the deepfakes is explicitly stated, and the resulting financial and epistemic harms are well-documented.

Scammers utilized generative AI to create realistic deepfakes of Australian health experts to promote unapproved diabetes supplements. While posing minor physical risks to patients and eroding trust in public health, the incident remains a localized consumer fraud issue with negligible direct national security implications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: Australia
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Near-term. Active ongoing scam campaign on social media requiring monitoring and response from regulators and platforms.
• Autonomy: Human-controlled. AI tools were directly controlled and deployed by human scammers to generate synthetic media.
• Novelty: Evolved capability. Represents an evolution of existing fraud methods, leveraging advanced generative AI and voice cloning to increase the realism and scale of scams.

Impact by dimension

• Physical security: Minor. Potential health risks to diabetes patients who might alter or stop prescribed medication due to misleading deepfake advertisements, though no direct physical casualties are reported.
• Information security: Minor. Use of AI-generated deepfakes of public figures and medical experts to spread misinformation, though motivated by financial fraud rather than state-sponsored information warfare.
• Sovereignty: Negligible. No significant threat to state authority, territorial control, or core government decision-making processes.
• Economic security: Minor. Individual financial losses from fraudulent sales of unapproved dietary supplements, representing localized consumer fraud rather than systemic economic threat.
• Societal stability: Minor. Erosion of public trust in medical professionals and health institutions due to convincing deepfake advertisements, though localized and manageable under consumer protection frameworks.