Inappropriate Gmail Smart Reply Suggestions

Google's Gmail Smart Reply tool was over-recommending the response "I love you" in situations where it was deemed innappropriate.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI system lacked the capability to understand social context, leading to inappropriate and robustly incorrect suggestions like 'I love you' in professional contexts.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Users relying on automated replies can lead to shallower human connections and dehumanized communication.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The issue was caused by the Smart Reply AI system unintentionally generating inappropriate suggestions due to unexpected outcomes of its training process, which was observed during early post-deployment rollouts and testing.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is an AI assistant that interacts with users to suggest text replies, falling under transparency obligations for AI systems interacting with humans.

AI system and alleged parties

• AI system: Smart Reply (Google)
• AI purpose: Writing Assistant; Predictive Typing
• Behaviour type: Assistant
• Alleged developer: Google
• Alleged deployer: Google
• Alleged harmed parties: Gmail Users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly exposed: 1400000000

Potential causes

Management

• Lack of Pre-deployment Checks: Features were tested internally but lacked early workplace safety guidelines.

Technology

• Frequent Phrase Bias: Model defaulted to common phrases like 'I love you' when signal was weak.
• Lack of Social Context: Neural network could not distinguish between personal and professional context.
• Semantic Duplication: Early algorithms suggested multiple replies with nearly identical meanings.

Data Inputs

• Unfiltered Training Data: Dataset included default signatures like 'Sent from my iPhone' as replies.
• Affectionate Tester Bias: Internal Google testers used 'I love you' frequently, biasing the prototype.

Human Factors

• Implicit Signature Usage: Users frequently sending default iPhone signatures skewed the training corpus.

Process and Methods

• Absence of SFW Filtering: Initial training lacked filters to block unprofessional or affectionate replies.
• Lack of Diversity Parameters: Early tuning did not enforce sufficient semantic distance between suggestions.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the Smart Reply feature's development, testing, and rollout, including specific anecdotes about the 'I love you' and 'Sent from my iPhone' suggestions. Multiple sources confirm the technical details of the LSTM neural networks used and the organizational response by Google.
• Ambiguities identified: There is slight ambiguity regarding whether any end-users actually sent 'I love you' to their bosses in production, or if the bug was entirely mitigated during internal testing and early beta phases.

A minor, unintentional software bug in Google's Smart Reply feature suggested inappropriate phrases like 'I love you' due to training data patterns. The issue was quickly mitigated by Google engineers prior to widespread impact and poses negligible national security risk.

• Overall national security impact: Negligible
• Scope: Multiple nations
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents a historical, resolved software bug with no active or imminent threat to national security.
• Autonomy: Human-controlled. The AI acts as a writing assistant, suggesting replies that require explicit human selection and action to send.
• Novelty: Established threat. Software bugs and unexpected training data biases in autocomplete and text prediction systems are well-established issues.

Impact by dimension

• Physical security: Negligible. No impact on physical security, critical infrastructure, or kinetic systems reported.
• Information security: Negligible. The incident involves minor, non-malicious autocomplete suggestions in a consumer email product with no information warfare or intelligence implications.
• Sovereignty: Negligible. No impact on state authority, electoral systems, or core government functions.
• Economic security: Negligible. No threat to strategic industries, financial systems, or technological competitive advantage; the issue was a minor product bug resolved internally.
• Societal stability: Negligible. No threat to social cohesion, civil liberties, or human rights; user concerns were limited to minor social awkwardness or perceived communication dehumanization.