AI-Generated OB-GYN Health Influencers on TikTok Used Fake Medical Credentials to Promote Dubious Advice

AI-generated avatars posing as OB-GYNs on TikTok, part of the so-called, crudely termed "Coochie Doctor" trend, have been falsely claiming years of experience while promoting dubious health advice. Many, including an avatar named "Violet," were created using the Captions app, which generates scripted AI influencers.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The incident involves the intentional creation of fake medical personas to deceive social media users and promote wellness products for financial and personal gain.

Additional risk subdomains

• 3.1 False or misleading information: The AI-generated avatars are used to spread unfounded and potentially dangerous medical advice to millions of social media users.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The risk is caused by human users who intentionally use the Captions app to generate deceptive medical personas and scripts to mislead the public for personal gain.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes AI-generated content such as deepfakes, which pose a moderate risk and require specific transparency obligations to ensure users are informed and aware of the AI's nature.

AI system and alleged parties

• AI system: Captions app (Captions)
• AI purpose: Deepfake Video Generation; Voice Generation
• Behaviour type: Tool
• Alleged developer: Captions
• Alleged deployer: Unknown TikTok users
• Alleged harmed parties: TikTok users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Minor
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Substantial
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes the creation and spread of deceptive synthetic videos impersonating medical professionals.

Directly caused: The Captions app was used to directly generate realistic AI videos of fake doctors delivering medically unfounded scripts.

Indirectly caused: These videos were uploaded to TikTok, reaching millions of views and generating tens of thousands of interactions.

Inferred additional harm: Numerous other accounts likely use similar synthetic avatars to spread deceptive or malicious content across various social media platforms.

Epistemic

Reported: The report explicitly describes epistemic harm through the spread of unfounded medical advice and the erosion of trust in online information.

Directly caused: The AI avatars fabricated medical authority, claiming over a decade of experience to deliver false health claims.

Indirectly caused: This led to thousands of users believing the fake doctors, with 36,000 comments on one video requesting recipes and wellness hub links, eroding the shared understanding of credible medical advice.

Inferred additional harm: The proliferation of such deepfakes at scale threatens to severely pollute the online information ecosystem and undermine public trust in legitimate medical professionals.

People affected

• Occurrences reported: 1
• People reportedly exposed: 3000000

Potential causes

Management

• Prioritizing Growth Over Safety: Captions app focuses on expanding user base rather than safety guardrails.
• Inadequate Risk Assessment: App developers failed to foresee misuse of avatars for medical scams.

Technology

• Realistic Generative AI Avatars: Captions app generates highly realistic human avatars like Violet.
• Text-to-Speech Video Generation: Users can easily generate talking videos by typing scripts into the app.
• Uncanny Visual Discrepancies: Avatars have stilted pauses, unblinking eyes, and audio-video lag.

Data Inputs

• Unverified Script Input: Users can input any script, including false medical claims, without checks.
• Cloned Human Likenesses: Companies clone real peoples likenesses to use as templates for avatars.

Human Factors

• Lack of Viewer Discernment: Many viewers cannot easily distinguish AI avatars from real doctors.
• Exploitation by Scammers: Malicious creators use AI to hawk wellness products and get comments.
• Trust in Social Media Advice: Users readily believe medical advice seen on their TikTok feeds.

Process and Methods

• Inadequate Platform Moderation: TikTok fails to filter out deepfake doctors spreading health misinformation.
• Lack of AI Content Labeling: Platforms do not automatically label AI-generated videos as artificial.
• Easy Video Production Process: One-click video creation allows creators to spam dubious content rapidly.

Regulatory Environment

• Absence of AI Impersonation Laws: No strong legal framework exists to ban AI impersonation of professionals.
• Lack of Medical Advice Oversight: Social media platforms lack regulation regarding medical advice dissemination.

Information quality

• Classification confidence: High
• Reason for confidence: The reports consistently describe the mechanics of the Captions app, the nature of the 'deepfake doctor' videos, and the public reaction. There is clear consensus across multiple sources (news.com.au, Daily Dot, Indy100, NY Post) regarding how the AI is being used and the specific medical claims being made.
• Ambiguities identified: None of significance; the role of the AI tool and the human creators is clearly delineated.
• Alternative interpretations: The incident could be viewed primarily as a moderation failure by TikTok rather than an AI safety issue, but the central role of synthetic media generation makes the AI safety classification highly appropriate.

The incident involves the deployment of consumer-grade generative AI to create realistic deepfake medical professionals on TikTok for commercial fraud. While it poses public health and epistemic risks by spreading misinformation, the national security impact is minor, representing an evolved capability for consumer fraud rather than a coordinated state-sponsored threat.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Individual social media users

Threat characteristics

• Imminence: Long-term. Reflects an ongoing, persistent consumer fraud threat rather than an active national security crisis requiring urgent governmental response.
• Autonomy: Human-controlled. The AI system functions purely as a tool, generating video and audio outputs based entirely on scripts and selections provided by human users.
• Novelty: Evolved capability. Represents an evolved capability where consumer-grade generative AI tools make high-quality video deepfakes accessible to low-skilled actors for fraud.

Impact by dimension

• Physical security: Negligible. No threats to critical infrastructure, kinetic systems, or national-level physical security. While false medical advice poses individual health risks, it does not constitute a national security physical threat.
• Information security: Minor. Involves AI-generated synthetic media (deepfakes) used for deceptive purposes on social media, but lacks state-sponsored backing or intent to target national institutions or intelligence capabilities.
• Sovereignty: Negligible. No impact on state authority, government decision-making, elections, or constitutional processes.
• Economic security: Negligible. Limited to low-level consumer fraud and unauthorized product promotion. No threat to strategic industries, financial systems, or national technological advantages.
• Societal stability: Minor. Erodes public trust in medical authorities and pollutes the information ecosystem with synthetic personas, representing a minor threat to societal epistemic stability.