Machine Personal Assistants Failed to Maintain Social Norms

During an experiment of software personal assistants at the Information Sciences Institute (ISI) at the University of Southern California (USC), researchers found that the assistants violated the privacy of their principals and were unable to respect the social norms of the office.

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information

The Electric Elves agents failed to maintain privacy boundaries in an office environment, leading to unauthorized sharing or leaking of sensitive personal schedule and location information among colleagues.

Additional risk subdomains

• 5.2 Loss of human agency and autonomy: The project faced challenges with adjustable autonomy, where agents had to dynamically adjust their level of decision-making, sometimes overriding or acting without explicit human consent.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The privacy and social norm violations were caused by the actions of the deployed Electric Elves agents, which was an unintentional outcome during their active post-deployment phase.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Risk Level 4: Minimal or No Risk. The Electric Elves system consists of software personal assistants used for routine office coordination and scheduling, which poses low or negligible risk to users and society.

AI system and alleged parties

• AI system: Electric Elves (University of Southern California)
• AI purpose: Team Coordination; Scheduling
• Behaviour type: Multi-agent
• Alleged developer: USC Information Sciences Institute
• Alleged deployer: USC Information Sciences Institute
• Alleged harmed parties: USC Information Sciences Institute

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Privacy

Reported: The report explicitly states that the deployment led to unanticipated observations and concerns about privacy in office environments.

Directly caused: The agents failed to maintain privacy boundaries, potentially sharing sensitive calendar, location, or availability data of the office members without proper consent.

Inferred additional harm: It is likely that the 12 users experienced minor privacy intrusions and disruptions to their daily office social dynamics due to the agents' sharing of information.

People affected

• Occurrences reported: 1
• People reportedly harmed: 12
• People reportedly exposed: 12

Potential causes

Management

• Overlooking Social Impacts: Project management focused on coordination, overlooking social and privacy risks.

Technology

• Flawed Adjustable Autonomy: Agents dynamically adjusted autonomy levels causing unexpected behaviors.
• Inadequate Privacy Protection: Agents tracked and shared user locations, violating office privacy.
• Lack of Social Norm Awareness: Agents lacked programming to respect subtle social norms in the office.

Human Factors

• User Privacy Concerns: Office members felt uncomfortable with constant agent tracking and sharing.
• Social Disruption: Users experienced disruptions due to agents violating workplace social norms.

Process and Methods

• Inadequate Risk Assessment: The deployment process did not anticipate privacy and autonomy challenges.
• Flawed Autonomy Design: Methods for adjusting agent autonomy dynamically were not fully matured.

Information quality

• Classification confidence: Medium
• Reason for confidence: The report clearly identifies the system, its deployment duration, and the specific areas of failure (privacy, adjustable autonomy, social norms). However, it is a high-level research summary and lacks specific anecdotes or technical details of individual privacy-violating events.
• Ambiguities identified: The exact nature of the privacy and social norm violations is not detailed, nor are the specific mechanisms of the adjustable autonomy failures.
• Alternative interpretations: The incident could be viewed primarily as an HCI design challenge rather than a privacy failure, given the focus on adjustable autonomy and social norms.
• Missing information: Specific examples of what information was leaked, how the agents violated social norms, and detailed user feedback or complaints.

An early academic deployment of software personal assistants in 2000 highlighted early socio-technical challenges in AI privacy and adjustable autonomy. The incident occurred entirely within a university research context and poses negligible national security risk.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident occurred in 2000 as a research project, representing early-stage academic study of agent autonomy rather than an active crisis.
• Autonomy: Human-supervised. The agents possessed adjustable autonomy, dynamically attempting to make decisions but remaining within a supervised academic research environment.
• Novelty: First-of-its-kind. One of the earliest real-world deployments of a multi-agent system managing adjustable autonomy and social norms in a human office environment.

Impact by dimension

• Physical security: Negligible. The incident occurred entirely within an academic office setting with no physical systems, critical infrastructure, or kinetic capabilities involved.
• Information security: Negligible. No classified information, state intelligence capabilities, or systematic information warfare campaigns were involved in this localized research trial.
• Sovereignty: Negligible. The project had no impact on state authority, territorial control, electoral systems, or core government decision-making processes.
• Economic security: Negligible. Minimal economic impact; this was a small-scale university research project with no strategic economic assets or critical technologies compromised.
• Societal stability: Negligible. While minor privacy issues occurred among approximately a dozen university staff members, there was no large-scale threat to social cohesion or human rights.