PimEyes's Facial Recognition AI Allegedly Lacked Safeguards to Prevent Itself from Being Abused

PimEyes offered its subscription-based AI service to anyone in the public to search for matching facial images across the internet, which critics said lacked public oversight and government rules to prevent itself from misuse such as stalking women.

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information

PimEyes allows users to upload a photo of any individual and correctly infer their identity and online presence across 900 million indexed images without their consent.

Additional risk subdomains

• 4.3 Fraud, scams, and targeted manipulation: The tool is actively exploited by malicious actors on forums like 4chan to stalk, dox, and compile humiliating dossiers on targeted female victims.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The primary risks and harms arise from the intentional deployment of the tool without safeguards by its developers and the intentional misuse of the tool by stalkers and anonymous users.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is a facial recognition application available to the general public, which falls under biometric identification and law enforcement-adjacent surveillance technologies.

AI system and alleged parties

• AI system: FaceNet, PimEyes (Google)
• AI purpose: Face Recognition; Image Search
• Behaviour type: Tool
• Alleged developer: PimEyes
• Alleged deployer: PimEyes
• Alleged harmed parties: internet users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Substantial
• Differential treatment: direct Negligible, indirect Minor
• Civil rights: direct Negligible, indirect Substantial
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Severe, indirect Severe
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: The report describes users having to pay 79.99 dollars per month to block their photos from showing up in search results.

Indirectly caused: Victims of non-consensual indexing must pay 79.99 dollars monthly for the PROtect package to secure their privacy.

Inferred additional harm: Many individuals likely incur ongoing financial losses paying subscription fees to protect their personal images from being searchable.

Malicious content

Reported: The report explicitly describes the creation and sharing of doxxing collages and stalking threads on anonymous forums.

Indirectly caused: Anonymous users on 4chan created and shared digital collages containing sensitive personal details, home addresses, and workplaces of targeted women.

Inferred additional harm: Numerous toxic and malicious doxxing dossiers are likely created and circulated across anonymous online communities using PimEyes search results.

Differential treatment

Reported: The report explicitly describes how the tool is disproportionately used to target and stalk young girls and women.

Indirectly caused: Young girls and women are disproportionately subjected to non-consensual tracking, stalking, and harassment on forums like 4chan.

Inferred additional harm: Systemic gender-based harassment and targeting of women is facilitated at scale due to the tool's availability.

Civil rights

Reported: The report explicitly describes concerns regarding the violation of basic privacy rights and the lack of consent for indexing personal images.

Indirectly caused: The tool indexes and makes searchable the faces of millions of individuals without their consent, violating their fundamental right to privacy.

Inferred additional harm: Millions of individuals globally have had their civil liberties and privacy rights compromised by being cataloged in a commercial surveillance database.

Privacy

Reported: The report explicitly describes extensive privacy violations where individuals' personal details, locations, and past photos are exposed.

Directly caused: The AI system indexes and matches faces across 900 million images, deanonymizing individuals without their consent.

Indirectly caused: Stalkers use the matched images to uncover victims' home addresses, workplaces, driver's licenses, and other highly sensitive personal information.

Inferred additional harm: Millions of individuals have had their expectation of privacy compromised by having their public and private histories linked via facial recognition.

Psychological

Reported: The report explicitly describes psychological distress and stalking experienced by individuals targeted using the tool.

Indirectly caused: Individuals experienced distress, feeling creeped out, and fear due to being stalked and doxxed on anonymous forums like 4chan.

Inferred additional harm: It is highly likely that hundreds of women targeted by stalkers using this tool have suffered significant anxiety, fear, and psychological trauma.

People affected

• Occurrences reported: 1
• People reportedly harmed: 2
• People reportedly exposed: 900000000

Potential causes

Management

• Prioritizing Profit Over Safety: Monetized advanced search and opt-out features despite abuse risks.
• Extreme Corporate Anonymity: Owners hide identities and corporate details in offshore havens.
• Lack of User Accountability: Management shifts blame for abuse entirely onto the end-users.

Technology

• Highly Accurate Matching Algorithm: Enables precise matching across different ages and head positions.
• Off-the-Shelf Open Source AI: Freely available tools like FaceNet lower barriers to build facial search.
• Web Scraping Bots: Spiders crawl the web, scanning and indexing faces as numerical code.

Data Inputs

• Unconsented Image Scraping: Scrapes 900 million images from across the web without user consent.
• Indexing of Sensitive Sites: Catalogues images on pornographic websites and social media.
• Lack of Data Verification: No verification of the source or ownership of indexed photos.

Human Factors

• Stalking and Harassment: Users utilize the tool to stalk women and compile personal details.
• Vigilante Investigations: Sedition hunters use the tool for crowdsourced policing without training.
• Circumvention of Terms: Users easily bypass terms of service to search for other people.

Process and Methods

• Flawed Self-Search Verification: Enforces self-search rule with just a single, easily bypassed checkbox.
• Pay-to-Block Model: Requires users to pay a monthly fee to block their photos from searches.
• One-at-a-Time Removal: Allows manual removal of only one image at a time unless paying.

Regulatory Environment

• Absence of National Regulation: No federal laws regulate public facial recognition use in the US.
• Jurisdictional Obscurity: Company registered in Seychelles to evade strict EU GDPR regulations.
• Lack of Public Oversight: No government rules control public access to powerful surveillance tools.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides detailed, first-hand accounts, security researcher testimonies, and specific examples of how the PimEyes tool is used and abused. The mechanics of the tool and its societal impacts are clearly documented.
• Ambiguities identified: The exact algorithms used by PimEyes are proprietary and undisclosed, and the company's ownership structure is obscured through a Seychelles shell company.
• Alternative interpretations: The company claims the tool is a boon for privacy by allowing self-monitoring, though evidence strongly points to its primary use as a surveillance and stalking tool.

PimEyes represents a significant advancement in democratized mass biometric surveillance, allowing anonymous users to deanonymize individuals globally. While its primary impacts are societal, affecting privacy and human rights through stalking and doxxing, its capability to track individuals poses long-term concerns for information security and personal privacy.

• Overall national security impact: Substantial
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global
• Alleged perpetrator: Anonymous individuals

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic privacy and surveillance concern rather than an immediate, acute national security crisis.
• Autonomy: Human-controlled. The AI system acts as a search utility, requiring human input to upload photos and human decisions to act on the generated search results.
• Novelty: Evolved capability. Represents a significant scaling and democratization of existing facial recognition and indexing capabilities, lowering the barrier to mass surveillance.

Impact by dimension

• Physical security: Negligible. No kinetic threats, physical attacks, or critical infrastructure compromises are reported in connection with this incident.
• Information security: Minor. The tool facilitates open-source intelligence gathering and deanonymization of individuals, which could be leveraged for targeted harassment or tracking, though no state-level intelligence compromise is documented.
• Sovereignty: Minor. The technology has minor intersections with government functions, such as identifying Capitol riot suspects and tracking political activists by foreign authorities.
• Economic security: Negligible. No significant threats to national economic stability, financial systems, or strategic technological competitive advantages are indicated.
• Societal stability: Substantial. The tool enables unregulated mass biometric surveillance and privacy violations on a global scale, facilitating systemic gender-based harassment, doxxing, and stalking.