Image Upscaling Algorithm PULSE Allegedly Produced Facial Images with Caucasian Features More Often

Image upscaling tool PULSE powered by NVIDIA's StyleGAN reportedly generated faces with Caucasian features more often, although AI academics, engineers, and researchers were not in agreement about where the source of bias was.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The upscaling algorithm performs with significantly lower accuracy and effectiveness for people of color compared to white individuals, resulting in unequal outcomes.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The system misrepresents individuals of color by generating white faces from their pixelated images, reinforcing racial biases.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The racial bias was an unexpected and unintentional outcome of the model's design and training data, which became apparent after the code was released publicly.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: The system is an image upscaling tool that generates synthetic, highly realistic human faces (deepfakes/synthetic media) using StyleGAN, which falls under transparency obligations for AI-generated content.

AI system and alleged parties

• AI system: PULSE
• AI purpose: Image Upscaling; Image Generation
• Behaviour type: Tool
• Alleged developer: Duke researchers
• Alleged deployer: Duke researchers
• Alleged harmed parties: people having non-Caucasian facial features

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment where the algorithm performs poorly on people of color compared to white people.

Directly caused: The PULSE algorithm consistently generated Caucasian features when processing low-resolution images of people of color, such as Barack Obama and Alexandria Ocasio-Cortez, while performing as expected on white faces.

Inferred additional harm: People of color using or represented by such upscaling algorithms face systemic exclusion and misrepresentation in computer vision applications.

Epistemic

Reported: The report describes how the tool generates realistic but completely fake high-resolution faces that do not represent the ground truth.

Directly caused: PULSE generated highly realistic but false high-resolution images of individuals, which could mislead people into thinking they are real photos of the subjects.

People affected

• Occurrences reported: 1
• People reportedly harmed: 3
• People reportedly exposed: 3

Potential causes

Management

• Neglecting Research Bias Norms: Lab leadership downplayed research-stage bias compared to deployed products.
• Lack of Diverse Leadership: Demographic homogeneity in AI leadership limits awareness of bias risks.

Technology

• StyleGAN Generative Bias: The underlying generative model inherently favors generating white faces.
• Search Algorithm Limitations: PULSE selection algorithm search path defaults to white faces in latent space.

Data Inputs

• Biased Training Dataset: StyleGAN was trained on Flickr data containing over 72% white faces.
• Biased Evaluation Dataset: CelebA dataset used for testing contains 90% white faces, masking bias.

Human Factors

• Lack of Developer Awareness: Undergraduate researchers were not initially thinking about demographic biases.
• Homogeneous Development Teams: Lack of diversity in development teams leads to overlooking bias risks.

Process and Methods

• Flawed Benchmarking Practices: Using a biased benchmark dataset like CelebA masked the model's biases.
• Lack of Downstream Bias Testing: Failing to test the tool on diverse non-white faces before public release.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, consistent, and detailed accounts of the technical mechanism of PULSE and StyleGAN, the specific inputs tested (Obama, Ocasio-Cortez), and the resulting demographic biases. The debate between researchers is well-documented, and the technical causes (dataset bias in StyleGAN and CelebA) are clearly identified.
• Ambiguities identified: There is some minor technical debate between Yann LeCun and other researchers about whether the bias is purely dataset-driven or also influenced by the search algorithm itself.
• Alternative interpretations: The incident could be interpreted purely as an academic research limitation rather than a harmful deployment, though its public release on GitHub makes it a post-deployment safety issue.

The public release of the PULSE image upscaling algorithm revealed significant racial bias, consistently generating Caucasian faces from low-resolution images of people of color. While the incident sparked critical academic and ethical debates regarding dataset bias and diversity in AI development, it posed negligible direct threats to national security, critical infrastructure, or sovereignty.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: No clear primary
• Alleged perpetrator: Duke University researchers

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing strategic and ethical concern regarding AI bias rather than an active, time-sensitive national security crisis.
• Autonomy: Human-controlled. The PULSE algorithm operates as a static tool requiring human input (low-resolution images) to generate outputs, with no autonomous decision-making or execution.
• Novelty: Evolved capability. While dataset bias is a known issue, the generation of highly realistic Caucasian faces from pixelated inputs of people of color represented an evolved manifestation of algorithmic bias.

Impact by dimension

• Physical security: Negligible. No physical threats, kinetic attacks, or critical infrastructure manipulation occurred as a result of this academic image upscaling tool.
• Information security: Negligible. While the tool generates synthetic facial features, there is no evidence of it being used for systematic information warfare, intelligence compromise, or hostile disinformation campaigns.
• Sovereignty: Negligible. No disruption to core government operations, electoral systems, or state sovereignty was reported; the use of political figures' images was for public testing purposes.
• Economic security: Negligible. The incident represents an academic and public debate on AI bias, with no direct threat to strategic industries, critical supply chains, or national technological security.
• Societal stability: Minor. The algorithm demonstrated systematic demographic bias by white-washing faces of people of color, highlighting broader ethical and civil rights concerns in computer vision, but without causing civil unrest.