Student-Developed Facial Recognition App Raised Ethical Concerns

TheFaceTag app, a social networking app developed and deployed within-campus by a student at Harvard raised concerns surrounding its facial recognition, cybersecurity, privacy, and misuse. This incident has been downgraded to an issue as it does not meet current ingestion criteria.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The reports detail various AI systems failing to perform reliably or safely under experimental conditions, such as GPT-3 suggesting suicide or Tesla Autopilot being vulnerable to adversarial stickers.

Additional risk subdomains

• 2.2 AI system security vulnerabilities and attacks: Tencent Keen Security Lab identified adversarial sample vulnerabilities in Tesla's Autopilot system.
• 1.2 Exposure to toxic content: An experimental GPT-3 medical chatbot suggested that a mock patient kill themselves.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Other

The underlying reports describe unintentional AI system vulnerabilities, limitations, and unexpected behaviors across various experimental or pre-deployment settings.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report includes systems like Tesla Autopilot (critical transport infrastructure) and facial recognition (TheFaceTag), which fall under High Risk (Level 2) applications, alongside chatbots (Level 3).

AI system and alleged parties

• AI system: GPT-3, Tesla Autopilot
• AI purpose: Chatbot; Autonomous Driving
• Behaviour type: unknown
• Alleged developer: Yuen Ler Chow
• Alleged deployer: Yuen Ler Chow
• Alleged harmed parties: TheFaceTag app users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 6

Potential causes

Management

• Premature Deployment: Management experimented with LLMs for clinical tasks before verifying safety.

Technology

• Inconsistent LLM Outputs: GPT-3 displayed highly inconsistent responses during medical triage tests.
• Lack of Safety Guardrails: The model lacked hard constraints to prevent recommending self-harm.

Data Inputs

• General Web Training Data: The model was trained on general web text rather than verified medical facts.

Human Factors

• Experimental Prompting: A mock patient tested the chatbot with highly sensitive mental health queries.

Process and Methods

• Inadequate Clinical Validation: The deployment lacked rigorous clinical safety testing and validation processes.

Regulatory Environment

• Absence of LLM Medical Standards: There were no regulatory standards governing the medical use of LLMs.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly outlines the administrative decisions to downgrade several incidents, providing explicit reasons for each (e.g., 'academic finding', 'projected harms', 'not deployed in the real world'). This makes the assessment of zero real-world harms highly certain.
• Ambiguities identified: None. The text is very clear about why each incident was downgraded.
• Alternative interpretations: One could interpret the Nabla GPT-3 medical chatbot experiment as a real-world incident of toxic content generation, but the text explicitly notes it was an experiment with a mock patient and not deployed, justifying the downgrade.

The incident database administrative action to downgrade several historical reports highlights that these events represented academic findings, humor, or projected risks rather than real-world national security threats. None of the evaluated incidents resulted in physical harm, critical infrastructure compromise, or information warfare.

• Overall national security impact: Negligible
• Scope: Unknown
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. These are historical academic findings and experiments with no active or immediate threat profile.
• Autonomy: Human-supervised. The systems involved, such as Tesla Autopilot and GPT-3, operate with human oversight or within controlled experimental environments.
• Novelty: Established threat. The incidents represent well-known academic evaluations of AI limitations and vulnerabilities rather than novel threat vectors.

Impact by dimension

• Physical security: Negligible. No actual physical harm or critical infrastructure disruption occurred; the Tesla Autopilot vulnerability was a projected harm identified in an academic study.
• Information security: Negligible. No active information warfare or intelligence compromise occurred; the GPT-3 op-ed and other language model outputs were experimental or academic in nature.
• Sovereignty: Negligible. No threats to state authority, electoral systems, or core government operations were reported in these downgraded incidents.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or economic manipulation occurred; these were academic or experimental findings.
• Societal stability: Negligible. No real-world societal instability or human rights violations occurred; the facial recognition app and GPT-3 medical experiments did not result in real-world deployment or harm.