Purportedly AI-Enhanced Arrest Image Reportedly Led Viewers to Mistake a Girl for a Boy in Metz, France

A purportedly clearer, AI-modified frame from an arrest video in Metz, France, reportedly showed additional facial and body hair and a different shirt pattern. Viewers cited it as evidence that the arrested person was a boy, although she was a 14-year-old girl. The altered image allegedly distorted public interpretation of the arrest footage.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI upscaling tools generated false and misleading visual details (such as changing a girl's appearance to look like a boy, changing bus numbers, and fabricating a suspect's face) which misled the public and law enforcement.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The upscaling tools lack the capability to accurately reconstruct original details from low-resolution inputs, instead inventing synthetic data.
• 5.1 Overreliance and unsafe use: Law enforcement and social media users overrelied on these tools for critical identification tasks they are not fit for.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by AI upscaling models generating synthetic, fabricated details (AI entity) unintentionally while attempting to enhance image resolution post-deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes the technology being used in 'law enforcement applications' by the Serbian police to generate images of a suspect, which is a high-risk domain under the Act.

AI system and alleged parties

• AI system: ChatGPT, unspecified AI upscaling tools (OpenAI)
• AI purpose: Image Upscaling; Identification
• Behaviour type: Tool
• Alleged developer: Synthetic media generation technology developers, Synthetic image generation technology developers, AI image enhancement system developers
• Alleged deployer: Synthetic media creators, Social media users
• Alleged harmed parties: Women and girls, Social media users, Minors, Girls, Epistemic integrity, Adolescents, Adolescent girls

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Minor
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes the spread of false accusations and misleading images on social media platforms like X and TikTok.

Directly caused: AI tools directly generated misleading images that fabricated details of real-world events.

Indirectly caused: Social media users spread these AI-generated images to make false claims, such as accusing police of sexual assault or falsely identifying protesters.

Civil rights

Reported: The report explicitly describes violations of human or civil rights, such as the public misidentification of a minor and the dissemination of false suspect images by police.

Indirectly caused: The civil rights of the 14-year-old girl and the Serbian suspect were compromised due to false public accusations and dissemination of inaccurate forensic evidence.

Inferred additional harm: Potential false arrests or wrongful investigations of innocent people who happen to match the AI-fabricated faces.

Psychological

Reported: The report does not explicitly quantify psychological harm, but the public misidentification and false accusations imply significant distress for the affected individuals.

Indirectly caused: The 14-year-old girl in Metz likely experienced distress and anxiety due to public misidentification and false accusations on social media.

Inferred additional harm: The suspect in Serbia and other misidentified individuals likely suffered psychological distress from being falsely accused or hunted based on fabricated AI images.

Epistemic

Reported: The report explicitly describes epistemic harm, detailing how AI-upscaled images generated false leads and eroded the ability of the public and police to rely on visual evidence.

Directly caused: AI tools fabricated synthetic details (changing genders, faces, objects, and numbers) that did not exist in reality.

Indirectly caused: The public and law enforcement were misled into believing false narratives, such as a girl being a boy or a suspect having a completely different face, eroding trust in visual evidence.

Inferred additional harm: Widespread erosion of trust in video and photographic evidence as upscaling tools become common, making it harder to establish shared facts in public discourse.

People affected

• Occurrences reported: 5
• People reportedly harmed: 5
• People reportedly exposed: 10000

Potential causes

Management

• Failure to Warn Users of Limits: AI developers do not warn users against using tools for forensic tasks.
• Lack of Police Oversight: Police management failed to verify the validity of AI-enhanced images.

Technology

• Synthetic Detail Generation: AI upscaling invents plausible but entirely synthetic details to fill gaps.
• Facial Feature Distortion: Neural networks alter facial traits, making individuals unrecognizable.
• Alteration of Key Evidence: Algorithms modify text, clothing, and remove critical physical evidence.

Data Inputs

• Low-Resolution Source Images: Extremely blurry source inputs force the AI to invent missing details.
• Lack of Original Sensor Data: Upscaling relies on statistical patterns rather than actual raw data.

Human Factors

• Misunderstanding of AI Limits: Users falsely assume upscaling retrieves real data instead of inventing it.
• Confirmation Bias on Social Media: Users share AI-altered images to quickly support their narratives online.
• Overreliance by Law Enforcement: Police release AI-enhanced photos, leading to public misidentification.

Process and Methods

• Inappropriate Tool Selection: Using generative upscaling for forensic identification is highly flawed.
• Lack of Verification Protocols: Media and users share enhanced images without checking the original source.

Regulatory Environment

• No Standards for AI Evidence: No regulatory guidelines exist for using AI-modified images in policing.
• Unregulated Public AI Tools: Powerful generative tools are public without forensic use warnings.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, detailed examples of AI upscaling tools fabricating details across multiple distinct events in France and Serbia. The technical explanation of how upscaling works (inventing synthetic details rather than recovering original data) is well-supported by experts and journalists.
• Ambiguities identified: The specific names of the AI upscaling software used by the public and the Serbian police are not explicitly named, though ChatGPT is mentioned as a comparison tool.
• Alternative interpretations: The incidents could be viewed purely as human error or misinformation campaigns, but the core issue is the technical limitation of generative AI upscaling when applied to forensic tasks.

AI-powered upscaling tools fabricated synthetic details on blurry footage, leading to public misidentifications in France and misleading law enforcement investigations in Serbia. While presenting minor risks to societal stability and government functions due to compromised evidence reliability, the overall national security impact remains low.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: France, Serbia
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding the reliability of digital evidence rather than an active, time-sensitive national security crisis.
• Autonomy: Human-controlled. AI tools were used as assistants to upscale images, with humans making the final decisions to disseminate the results.
• Novelty: Evolved capability. Represents a significant advancement in image manipulation where AI generates highly plausible but completely fabricated details.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety threats were targeted or affected in these incidents.
• Information security: Minor. While the tools generated false visual information, there is no evidence of coordinated state-sponsored information warfare or compromise of classified intelligence.
• Sovereignty: Minor. Law enforcement operations in Serbia were temporarily hindered by relying on inaccurate AI-generated suspect images, representing a minor impact on government functions.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or economic impacts were reported.
• Societal stability: Minor. The incidents led to localized public misidentification, false accusations, and minor civil rights concerns, but did not threaten large-scale societal stability.