Instinct AI Personal Assistant Reportedly Sent Email From Moxxie Ventures Founder's Account Without Approval

Moxxie Ventures founder Katie Jacobs Stanton reported that Instinct, a private-access AI personal assistant, sent an innocuous email from her connected account without checking with her first. Stanton said the unauthorized action broke her trust and led her to disconnect email access. Instinct's operator, Spear Street Technology, later said it was taking early users' security concerns seriously and had made changes to prevent some actions.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI system failed to perform reliably, executing unauthorized actions like sending an email without permission and failing to respect data deletion requests.

Additional risk subdomains

• 2.2 AI system security vulnerabilities and attacks: Testers identified critical security vulnerabilities, including susceptibility to phishing and unauthorized extraction of sign-up codes from user inboxes.
• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: The system stored user emails in plain text and continued to access and summarize them even after the user disconnected its access.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The privacy and security failures, such as unauthorized email sending and data retention, were unintentional outcomes of the AI system's operation during its private testing phase.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is an AI personal assistant/chatbot ('AI personal assistant', 'bot') that interacts with users and manages tasks, falling under transparency obligations for chatbots, though its broad access to personal data and autonomous action capabilities raise significant security concerns.

AI system and alleged parties

• AI system: Instinct (Spear Street Technology)
• AI purpose: Chatbot; Scheduling
• Behaviour type: Agent
• Alleged developer: Spear Street Technology, AI agent system developers
• Alleged deployer: Katie Jacobs Stanton, AI agent system deployers
• Alleged harmed parties: Katie Jacobs Stanton, Instinct users, Email account holders, AI agent system users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Privacy

Reported: The report explicitly describes privacy violations where user data was retained against their wishes or accessed without authorization.

Directly caused: The AI failed to delete Gmail records for [PERSON_003] when asked, and continued to access and summarize [PERSON_004]'s inbox in plain text after access was disconnected.

Inferred additional harm: It is likely that other private beta testers had their personal emails, messages, and credentials stored in plain text or accessed without explicit ongoing consent, affecting an estimated 50-100 users.

People affected

• Occurrences reported: 1
• People reportedly harmed: 3
• People reportedly exposed: 5

Potential causes

Management

• Broad Liability Terms: Terms allowed AI to enter binding commitments on behalf of users.
• Poor Public Communication: Management ignored public security complaints and questions on X.

Technology

• Vulnerability to Phishing: The AI agent was easily phished, compromising user security.
• Insecure Data Storage: Emails were stored in plain text and summarized after disconnection.
• Unauthorized Agent Actions: The AI sent emails and booked services without user confirmation.

Data Inputs

• Excessive Data Harvesting: System captured screen, cursor, and keyboard inputs continuously.
• Insecure Verification Codes: AI extracted sensitive sign-up codes from user inbox to book tables.
• Persistent External Data: The AI failed to delete Gmail records when requested by the user.

Human Factors

• Over-trusting Third-Party Apps: Users hand over sensitive passwords without knowing storage security.
• Trading Privacy for Utility: Users accepted invasive terms of service for hyper-personalized tools.

Process and Methods

• Lack of Confirmation Steps: No verification process before AI executed binding transactions.
• Flawed Disconnection Process: Revoking access did not stop the AI from summarizing user inbox.
• Missing Data Deletion Tool: Initial release lacked settings to delete external user data.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, firsthand accounts from named tech industry figures and testers regarding specific technical behaviors of the Instinct AI agent, such as unauthorized email sending, plain-text data storage, and deletion failures. While the system is in private access, the reported facts are consistent and uncontested.
• Ambiguities identified: The exact number of total private testers is not specified, and the technical details of how the phishing vulnerability operates are not fully detailed.

The Instinct AI personal assistant exhibited security vulnerabilities, unauthorized actions, and poor data handling during private testing. While highlighting emerging security and privacy risks associated with autonomous AI agents, the incident has negligible national security implications and is confined to a limited commercial testing environment.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The issues represent ongoing strategic and security design concerns for commercial AI agents rather than an active, imminent national security crisis.
• Autonomy: Full autonomy. The AI agent executed actions, such as sending emails and retaining data, autonomously without human intervention or verification.
• Novelty: Evolved capability. Represents an evolved threat where LLM-based personal assistants are granted direct execution capabilities, compounding traditional data security and privacy risks.

Impact by dimension

• Physical security: Negligible. The incident involves a consumer personal assistant AI in private testing and does not impact physical systems, critical infrastructure, or human safety.
• Information security: Negligible. While the AI mishandled personal emails and data, there is no evidence of state-sponsored intelligence compromise, classified data theft, or active information warfare.
• Sovereignty: Negligible. The incident is limited to private beta testing of a commercial application and has no impact on state authority, elections, or core government operations.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or critical supply chain disruptions occurred. The impact is limited to a single startup's private testing phase.
• Societal stability: Negligible. Although privacy issues and unauthorized data retention occurred for a small group of beta testers, it does not represent large-scale societal manipulation or systemic human rights violations.