Instinct AI Personal Assistant Reportedly Summarized an AI Product Founder's Retained Emails After Google Access Was Disconnected

AI product founder Claire Vo reported that three hours after she disconnected Instinct from her Google account, the personal AI assistant summarized previously indexed emails. Vo said the Google connector was off, but Instinct had retained copies for later searching; reporting indicates the last stored email predated disconnection. The episode involved the agent acting on retained email data after Google access was revoked, not confirmed continued access to Vo's Gmail account.

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information

The primary issue involves the AI system retaining copies of private emails in plain text and failing to delete user records even after access was revoked, directly compromising user privacy.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI system exhibited a lack of robustness by performing unauthorized actions, such as sending an email on a user's behalf without prior confirmation.
• 2.2 AI system security vulnerabilities and attacks: Testers identified that the system was highly vulnerable to phishing and could easily pull sensitive sign-up codes from user inboxes.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The privacy and security risks arose from the AI system's actions, such as retaining data after disconnection and sending unauthorized emails, which were unintentional outcomes of its operation.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is described as an 'AI personal assistant' and a 'chatbot' that interacts with users to perform tasks, which falls under the category of chatbots requiring transparency under the EU AI Act.

AI system and alleged parties

• AI system: Instinct
• AI purpose: Chatbot; Scheduling
• Behaviour type: Agent
• Alleged developer: Spear Street Technology, AI agent system developers
• Alleged deployer: Spear Street Technology, Claire Vo, AI agent system deployers
• Alleged harmed parties: Privacy, Instinct users, Email account holders, Claire Vo, AI agent system users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Privacy

Reported: The report explicitly describes privacy violations where the AI retained user emails in plain text after access was disconnected and failed to delete Gmail records upon request.

Directly caused: The AI system retained copies of a user's emails in plain text and continued summarizing the inbox after access was revoked. Another user found the system failed to delete Gmail records when requested.

Inferred additional harm: Since the app is in private testing with multiple testers, it is likely other testers had their private emails and data retained without their active consent or knowledge.

People affected

• Occurrences reported: 4
• People reportedly harmed: 4
• People reportedly exposed: 100

Potential causes

Management

• Growth Over Security Focus: Prioritized rapid deployment of autonomous features over safety safeguards.
• Unfavorable Liability Terms: Shifted all transaction dispute liabilities entirely onto the users.
• Delayed Public Response: Management kept a low profile and did not address early public concerns.

Technology

• Vulnerability to Phishing: The agent was easily phished, leading to user account deletion.
• Insecure Plain Text Storage: Emails were stored in plain text for searches even after disconnection.
• No Action Confirmation System: The system sent emails on behalf of users without checking first.

Data Inputs

• Intrusive Device Data Collection: Collected screen captures, cursor movements, and keyboard inputs.
• Persistent Data Retention: Retained copies of emails and external inputs after disconnection.
• Default Model Training Use: Personal data was used for model training by default without consent.

Human Factors

• Unaware of Storage Risks: Users handed over passwords without knowing how they would be stored.
• Lack of Opt-Out Awareness: Users did not realize they had to manually opt out of data training.

Process and Methods

• Flawed Deletion Process: Lacked a tool to delete external Gmail records when requested by users.
• No Verification Workflows: Allowed binding commitments and purchases without explicit user confirmation.
• Silent Background Processing: Continued summarizing inboxes after email access was disconnected.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from TechCrunch and Norton provide clear, consistent details about the AI system's functionality, its terms of service, and specific user experiences highlighting privacy and security issues. There is little ambiguity about what the system did (retaining data, sending unauthorized emails) and how users reacted.
• Ambiguities identified: The exact number of private beta testers is not specified, and the technical details of how the phishing vulnerability works are not fully explained.
• Alternative interpretations: None. The reports consistently frame the incident as a privacy and security risk associated with a highly autonomous AI agent.

During private beta testing of the Instinct AI personal assistant, users discovered privacy and security flaws, including data retention after deletion requests and unauthorized email sending. This incident represents standard commercial software security and privacy challenges with negligible implications for national security.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The issues are confined to a private beta testing phase of a commercial application, posing no immediate national security threat.
• Autonomy: Human-supervised. The AI assistant performed tasks like sending emails and summarizing inboxes autonomously, though designed to operate under user oversight.
• Novelty: Established threat. Data retention failures, plain-text storage, and unauthorized actions by software agents are well-established security and privacy issues.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety were impacted or threatened by the private testing of this personal assistant.
• Information security: Negligible. While the system exhibited security vulnerabilities like plain-text storage and phishing susceptibility, there is no evidence of intelligence compromise or information warfare.
• Sovereignty: Negligible. The incident involved a private beta test of a commercial personal assistant and did not impact government operations or state sovereignty.
• Economic security: Negligible. No strategic technology theft or financial system attacks occurred; the incident is limited to security flaws in a commercial startup's beta product.
• Societal stability: Negligible. Privacy issues and unauthorized actions affected a small group of private beta testers, presenting no threat to broad societal stability or mass human rights.