Alleged Deepfake of New Zealand Endocrinologist Reportedly Promotes Misleading Diabetes Claim

A video circulated on social media that appeared to feature University of Otago endocrinologist Sir Jim Mann allegedly promoting a hemp gummy product for diabetes patients and urging them to stop using metformin. The video was reportedly generated using AI and has been described as spreading manipulated visuals and speech presenting false medical advice.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The incident involves the unauthorized use of public likenesses (deepfake video and voice clones) to deceive consumers for financial gain through fake medical endorsements.

Additional risk subdomains

• 3.1 False or misleading information: The deepfake video generated and spread highly dangerous medical misinformation, falsely advising diabetes patients to stop taking metformin.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was initiated by human malicious actors who intentionally deployed AI deepfake tools post-deployment to perpetrate a commercial scam.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3. Limited Risk: The report describes AI-generated content such as deepfakes ('AI is so clever that they could manipulate my mouth to be looking as if I was saying those words').

AI system and alleged parties

• AI system: None named
• AI purpose: Deepfake Video Generation; Voice Generation
• Behaviour type: Tool
• Alleged developer: Unknown deepfake technology developers
• Alleged deployer: Unknown scammers
• Alleged harmed parties: New Zealand general public, Jim Mann, Diabetes patients

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the reports describe a deepfake video and voice clone used for scams.

Directly caused: A deepfake video of Dr. [PERSON_002] and a voice clone of Dr. Brems were created and distributed on Facebook and Instagram.

Inferred additional harm: Additional promotional materials and copies of the video may have been shared across other social media platforms.

Psychological

Reported: Yes, the report describes Dr. [PERSON_002] feeling 'terrible' and being 'bombarded' with messages, causing distress.

Directly caused: Dr. [PERSON_002] experienced distress and anxiety due to the unauthorized use of his likeness to promote dangerous medical advice.

Inferred additional harm: Other impersonated professionals (like Dr. Brems) and misled patients likely experienced distress and anxiety upon realizing they were scammed.

Epistemic

Reported: Yes, the reports describe fake news videos and false stories circulating on social media.

Directly caused: The deepfake video falsely claimed that a prominent endocrinologist recommended stopping metformin in favor of alternative natural products.

Inferred additional harm: The incident contributes to a broader erosion of trust in medical professionals and public health advice on social media.

People affected

• Occurrences reported: 2
• People reportedly harmed: 2
• People reportedly exposed: 1000

Potential causes

Management

• Inadequate Ad Screening: Platform management failed to screen high-risk medical ads effectively.

Technology

• AI Deepfake Video Generation: AI was used to manipulate the doctor's mouth and clone his voice.
• Voice Cloning Technology: AI cloned the podiatrist's voice to fake a medical recommendation.

Data Inputs

• Publicly Available Likeness: Public media of the doctor was used to train the deepfake model.

Human Factors

• Overreliance on Authority: People trusted the video because it featured a known medical specialist.
• Lack of Digital Literacy: Viewers could not distinguish a highly realistic deepfake from a real clip.

Process and Methods

• Weak Social Media Moderation: Facebook failed to detect and block the deceptive deepfake ad quickly.
• Re-upload of Removed Content: The video occasionally resurfaced after initial removal.

Regulatory Environment

• Lack of Deepfake Regulation: Absence of strong legal frameworks to prevent AI impersonation of doctors.

Information quality

• Classification confidence: High
• Reason for confidence: The reports clearly describe the nature of the deepfake scam, the platforms used, and the reactions of the affected individuals. The boundary rules clearly resolve the classification of public likeness exploitation under 4.3 rather than 2.1.
• Ambiguities identified: The specific AI tools used to generate the deepfakes are not named.

The incident involves the use of AI-generated deepfakes and voice clones to impersonate medical professionals for commercial scams in New Zealand and the US. While posing risks to public health trust and individual consumers, the incident has minor direct national security implications and is manageable through standard law enforcement and regulatory procedures.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: New Zealand, United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing, strategic concern regarding the proliferation of synthetic media scams rather than an active national security crisis.
• Autonomy: Human-controlled. AI tools were used as static generation instruments controlled and deployed entirely by human operators to execute the scam.
• Novelty: Evolved capability. Deepfakes and voice cloning for commercial scams are established threats, though applying them to local medical authorities to bypass health regulations represents an evolved tactic.

Impact by dimension

• Physical security: Negligible. No kinetic threats or infrastructure compromise. While misleading medical advice poses individual health risks, it does not threaten national critical infrastructure or physical security.
• Information security: Minor. Demonstrates the use of AI-enhanced synthetic media to spread highly convincing disinformation, though currently limited to commercial fraud rather than state-sponsored intelligence operations.
• Sovereignty: Negligible. No impact on state authority, electoral systems, or core government decision-making processes.
• Economic security: Negligible. Limited to localized consumer fraud and does not threaten strategic industries, financial systems, or national technological advantage.
• Societal stability: Minor. Contributes to the erosion of public trust in medical professionals and health institutions, representing a minor threat to societal information integrity.