Attackers Reportedly Deployed Simulated AI Support Chatbot to Trick Instagram Business Users into Adding Malicious 2FA Login

A phishing campaign has reportedly been impersonating Meta support using a fake chatbot interface to hijack Instagram Business accounts. Victims received emails claiming ad violations and were directed to a fraudulent site mimicking Meta's support. There, a simulated chatbot instructed users to add the attacker’s Authenticator app as a secure login method, enabling account takeover. It remains unclear whether the chatbot used AI or was human-operated via a bot-like interface. See editor's note.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The threat actors used a fake tech support chatbot and deceptive social engineering tactics to defraud users and hijack their Meta Business accounts for personal gain.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by human threat actors intentionally deploying a malicious phishing campaign and fake chatbot to hijack user accounts.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes a chatbot, which falls under Risk Level 3 due to transparency obligations, though its deployment here is entirely malicious.

AI system and alleged parties

• AI purpose: Chatbot
• Behaviour type: Tool
• Alleged developer: Unknown generative AI developers
• Alleged deployer: Unknown scammers, Scammers impersonating Meta support, Phishers
• Alleged harmed parties: Meta users, Instagram users, Instagram Business users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Privacy

Reported: The report explicitly describes attempts to obtain sensitive personal and business information, including credentials and 2FA access.

Directly caused: The fake chatbot and setup guide attempted to collect names, business emails, Facebook passwords, and screenshots of personal information pages.

Inferred additional harm: Victims who followed the instructions had their account credentials and 2FA tokens compromised, leading to unauthorized access to their private business data.

Epistemic

Reported: The report explicitly describes the use of deceptive information, such as fake suspension notices and a fraudulent support chat, to mislead users.

Directly caused: The campaign generated fake alerts claiming users violated advertising laws and GDPR to create a false sense of urgency.

Inferred additional harm: Such campaigns erode general trust in official communications from platforms like Meta and Instagram.

People affected

• Occurrences reported: 1

Potential causes

Management

• Insufficient Security Training: Lack of employee training on social engineering and fake support chatbots.

Technology

• Fake Tech Support Chatbot: Malicious automated chatbot used to mimic official support and steal data.
• Abuse of Salesforce Platform: Attackers leveraged legitimate Salesforce email systems to bypass spam filters.
• Lookalike Domain Hosting: Fraudulent domain businesshelp-manager hosted to mimic Meta Business site.

Data Inputs

• Malicious Screenshot Requests: Attackers collected account screenshots to screen and target high-value victims.
• Phished User Credentials: Victims inputted sensitive login details and passwords into fraudulent forms.

Human Factors

• Urgency and Fear of Suspension: Fear of losing advertising capabilities drove hasty, unsafe decision-making.
• Lack of URL Verification: Users failed to verify domain names in browser address bars before acting.
• Trust in Automated Chat: Interactive nature of the fake chatbot created a false sense of security.

Process and Methods

• Flawed 2FA Setup Process: Users manually added attacker's authenticator app under 'System Check' guise.
• Inadequate Email Verification: Lack of strict validation processes allowed spoofed sender addresses.

Regulatory Environment

• Weak Lookalike Domain Controls: Lack of strict regulations on registering domains mimicking major brands.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide a highly detailed, step-by-step analysis of the phishing campaign's mechanics, including the email templates, landing pages, chatbot dialogues, and 2FA hijacking methods. While the exact number of victims and the specific technical architecture of the chatbot are not detailed, the overall flow and intent of the campaign are extremely clear.
• Ambiguities identified: The reports do not specify whether the 'fake tech support chatbot' uses generative AI or is a simple rule-based decision tree, nor do they provide statistics on the number of successful compromises.
• Alternative interpretations: The chatbot could be interpreted as a simple non-AI automated script rather than an AI system, which would place this outside the scope of traditional AI incidents.

A sophisticated cybercrime campaign utilized a fake tech support chatbot and social engineering to hijack Meta Business and Instagram accounts. While representing an evolved cyber threat to individual businesses and commercial entities, the incident has negligible direct national security implications.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing cybersecurity threat and trend in cybercrime rather than an active national security crisis.
• Autonomy: Human-controlled. The chatbot operates as a highly scripted tool deployed and directed by human threat actors to facilitate social engineering.
• Novelty: Evolved capability. Represents an evolution of traditional phishing techniques by integrating automated chat interactions and deceptive 2FA setup guides.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety threats were involved in or affected by this commercial phishing campaign.
• Information security: Negligible. The campaign targeted commercial social media business accounts for account hijacking, with no indications of state-sponsored intelligence compromise or strategic information operations.
• Sovereignty: Negligible. No government operations, electoral systems, or state sovereignty functions were targeted or disrupted by this incident.
• Economic security: Minor. While the hijacking of business accounts causes financial loss to individual businesses, it does not pose a systemic threat to national financial stability or strategic industries.
• Societal stability: Negligible. No threats to societal stability, civil liberties, or large-scale human rights were observed in this cybercrime campaign.