Google AI Overviews and ChatGPT Reportedly Cited Fraudulent Cruise Hotline, Allegedly Enabling Successful Scam

Google's AI Overviews allegedly surfaced a fraudulent customer service number for Royal Caribbean. Consumer Alex Rivlin reportedly called the number, spoke with an impostor, and provided his credit card, later incurring $768 in fraudulent charges. The same number reportedly appeared in ChatGPT results and was linked to Disney and Princess cruises, purportedly showing how scammers are manipulating AI outputs to amplify longstanding hotline scams.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI systems inadvertently surfaced and spread a fraudulent customer service number as an authoritative answer, leading a user to believe it was legitimate and suffer financial loss.

Additional risk subdomains

• 4.3 Fraud, scams, and targeted manipulation: Scammers manipulated the AI's data sources to facilitate a financial scam.
• 7.3 Lack of capability or robustness: The AI systems lacked the robustness to detect and filter out adversarial search engine optimization and data poisoning.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by Google's AI Overviews and ChatGPT unintentionally surfacing fraudulent phone numbers post-deployment due to data manipulation by external scammers.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The systems involved are search assistants and chatbots (Google AI Overviews and ChatGPT) which generate content and interact with users, falling under transparency obligations of Risk Level 3.

AI system and alleged parties

• AI system: ChatGPT, Google AI Overviews (Google, OpenAI)
• AI purpose: Question Answering; Content Search
• Behaviour type: Assistant
• Alleged developer: OpenAI, Google
• Alleged deployer: Unknown scammers impersonating Royal Caribbean, Unknown scammers impersonating Princess Cruises, Unknown scammers impersonating Disney Cruise Line, Unknown scammers
• Alleged harmed parties: Royal Caribbean customers, Royal Caribbean, Princess Cruises customers, Princess Cruises, General public, Disney Cruise Line customers, Disney Cruise Line, Alex Rivlin

Harm severity

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Epistemic

Reported: Yes, the report describes AI-driven misinformation where Google AI Overviews and ChatGPT surfaced a fraudulent customer service number as authoritative truth.

Directly caused: Google AI Overviews and ChatGPT directly generated and displayed false customer service numbers to users searching for cruise line contact information.

Indirectly caused: This led to the erosion of user trust in AI-generated search results and search engines as reliable gateways to information.

Inferred additional harm: It is highly likely that many other search queries for various businesses have been similarly poisoned, leading to widespread exposure to false contact information across the web.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Prioritizing Speed Over Safety: Google deployed AI Overviews for phone queries without adequate vetting.

Technology

• AI Overviews Hallucination: AI generated responses presented unverified phone numbers as authoritative.
• Susceptibility to Manipulation: Search and AI models easily manipulated by SEO spam techniques.

Data Inputs

• Unverified Web Scraping: AI models ingested fake contact numbers from unvetted forums and blogs.
• Lack of Vetted Databases: Systems failed to cross-reference phone numbers with official databases.

Human Factors

• Overreliance on AI Answers: Victim trusted Google AI Overviews as highly authoritative.
• Convincing Social Engineering: Scammers provided persuasive explanations and waived fees over the phone.

Process and Methods

• Inadequate Content Filtering: Search algorithms failed to detect and filter out coordinated scam spam.
• Delayed Information Update: AI models take too long to update after abusive content is removed.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides a clear, first-hand account of the victim's experience, verified by search analysts who replicated the issue on both Google AI Overviews and ChatGPT. The mechanics of the scam and the AI's role are well-documented.
• Alternative interpretations: The incident could be viewed purely as a traditional SEO spam/scam issue, but the integration of these numbers into AI-generated 'answers' (AI Overviews) represents a distinct AI safety failure.

Scammers successfully exploited Google AI Overviews and ChatGPT through data poisoning, leading the systems to autonomously surface fraudulent customer service numbers. While representing a novel evolution of SEO fraud that erodes trust in AI search utilities, the national security impact remains minor and is manageable under standard fraud prevention procedures.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Reflects an ongoing strategic vulnerability in AI data ingestion rather than an imminent national security crisis.
• Autonomy: Full autonomy. The AI models autonomously ingested poisoned web data and generated fraudulent outputs without real-time human verification.
• Novelty: Evolved capability. Represents an evolution of traditional SEO spam and phishing techniques, adapted to exploit modern AI-driven search and synthesis engines.

Impact by dimension

• Physical security: Negligible. No physical systems, infrastructure, or human safety elements were impacted or threatened by this incident.
• Information security: Negligible. The incident involves commercial fraud and search engine manipulation rather than state-sponsored disinformation or intelligence compromise.
• Sovereignty: Negligible. No government systems, electoral processes, or core state functions were targeted or disrupted.
• Economic security: Minor. Represents a minor economic security issue where consumer trust in major AI search utilities is exploited for financial fraud, though individual losses remain low.
• Societal stability: Negligible. There is no indication of threats to societal stability, civil liberties, or large-scale social manipulation.