Alleged Misuse of PicSo AI for Generating Inappropriate Content Emphasizing "Girls"

PicSo AI, which appears to be getting advertised by Meta over Instagram, is allegedly being used for generating inappropriate content with an emphasis on "girls." This raises concerns about the misuse of generative AI technologies for creating offensive and potentially sexually explicit material that could be used for nefarious and criminal purposes.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The incident involves an AI image generator being advertised and used to create sexually explicit content, which constitutes exposure to toxic and inappropriate content.

Causal factors

• Entity: AI
• Intent: Intentional
• Timing: Post-deployment

The risk is caused by the PicSo AI system generating explicit content, which is an intentional capability of the deployed application.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is an AI image generator producing synthetic content, which falls under transparency obligations for AI-generated content.

AI system and alleged parties

• AI system: PicSo AI
• AI purpose: Image Generation; Visual Art Generation
• Behaviour type: Tool
• Alleged developer: PicSo AI
• Alleged deployer: Meta, Instagram
• Alleged harmed parties: Potentially exploited groups, General public, Consumers

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and inappropriate content being promoted and generated by PicSo AI.

Directly caused: The advertisement promoted an AI tool designed to generate suggestive or sexually explicit content featuring 'girls'.

Inferred additional harm: The tool is likely being used by numerous users to generate large quantities of sexually explicit or inappropriate images.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Prioritizing Ad Revenue Over Safety: Ad platforms host risky ads to maximize revenue from AI developers.
• Exploitative App Monetization: PicSo developers explicitly targeted suggestive themes for user acquisition.

Technology

• Unrestricted Generative AI Model: AI model allows generation of highly explicit or suggestive content of girls.
• Flawed Ad Targeting Algorithms: Algorithms served inappropriate AI app ads alongside mainstream user feeds.

Data Inputs

• Unfiltered Training Data: Training data likely included sexualized images of young-looking individuals.

Human Factors

• User Demand for Explicit Content: Users actively seek and generate suggestive content using AI tools.

Process and Methods

• Weak Ad Moderation Processes: Meta failed to detect and block inappropriate AI app advertisements.
• Inadequate AI Safety Guardrails: PicSo lacked robust guardrails to prevent generating suggestive content.

Regulatory Environment

• Unclear AI Pornography Laws: Lax or ambiguous legal frameworks regarding AI-generated suggestive content.

Information quality

• Classification confidence: Medium
• Reason for confidence: The report is a single social media post expressing personal concern about an advertisement. While the core complaint about PicSo AI's marketing is clear, there is no independent verification of the app's outputs or legal status.
• Ambiguities identified: The term 'girls' is ambiguous as it could refer to adult women styled in a certain way or actual minors, and the reporter notes they cannot verify the legality of the content.
• Alternative interpretations: The application may be a standard anime-style image generator whose suggestive marketing is being interpreted as targeting minors.
• Missing information: Technical details of the AI model, verification of the actual images generated, and any official regulatory or legal investigations into PicSo AI.

A social media user reported that the PicSo AI image generator is being advertised on Instagram and potentially used to create sexually explicit content depicting minors. While presenting serious ethical and law enforcement concerns regarding child sexual exploitation and abuse (CSAE), the incident has minor direct implications for national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The issue represents an ongoing regulatory and law enforcement challenge rather than an immediate national security crisis.
• Autonomy: Human-controlled. The AI tool generates images based on direct user prompts, requiring human initiation and control.
• Novelty: Established threat. The misuse of generative AI models to create sexually explicit or inappropriate content is a well-documented and established threat.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic targeting mechanisms were impacted or involved in this incident.
• Information security: Negligible. There is no indication of state-sponsored information warfare, intelligence compromise, or classified data theft.
• Sovereignty: Negligible. Core government operations, electoral systems, and state sovereignty were not disrupted or threatened.
• Economic security: Negligible. The incident does not involve threats to financial systems, strategic technology theft, or supply chain vulnerabilities.
• Societal stability: Minor. While the potential generation of child sexual abuse material (CSAM) is a severe legal and human rights concern, the direct national security impact on societal stability remains minor.