Hiring Algorithms Provided Invalid Positive Results for Interview Responses in German

Two AI interview softwares provided positive but invalid results such as "competent" English proficiency and high match percentage for interview responses given in German by reporters.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI systems failed to perform reliably under varying conditions, such as processing German inputs or ignoring irrelevant visual features like glasses and backgrounds, leading to invalid assessments.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The systems unfairly represented candidates' traits based on irrelevant features like intonation, video backgrounds, and glasses, which can lead to biased hiring outcomes.
• 5.1 Overreliance and unsafe use: Employers are deploying and relying on these unvalidated AI tools for high-stakes hiring decisions despite their lack of scientific validity.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The failures in candidate evaluation and language processing were caused by the deployed AI systems producing unexpected and invalid outputs when presented with German inputs.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes AI systems used in recruitment and employment evaluation ('AI-powered interview software claims to help employers sift through applications'), which falls under 'Employment and worker management systems, such as AI used in recruitment' in the High Risk category.

AI system and alleged parties

• AI system: Retorio
• AI purpose: Application Evaluation; Automatic Skill Assessment
• Behaviour type: Assistant
• Alleged developer: MyInterview, Curious Thing
• Alleged deployer: MyInterview, Curious Thing
• Alleged harmed parties: job candidates using MyInterview, job candidates using Curious Thing, employers using MyInterview, employers using Curious Thing

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment of candidates based on irrelevant factors like video backgrounds and accessories.

Directly caused: Retorio's algorithm directly assessed candidates differently when they used different video backgrounds and accessories, like glasses, during their interviews.

Indirectly caused: Job seekers who wear glasses or have different home environments may be unfairly filtered out or scored lower by the automated systems.

Inferred additional harm: Systemic bias against certain demographic groups, such as individuals with visual impairments or lower socioeconomic status, is highly likely given the reliance on these unvalidated visual and auditory cues.

People affected

• Occurrences reported: 1
• People reportedly exposed: 1

Potential causes

Management

• Prioritizing Efficiency over Accuracy: Firms adopt AI to sift applications quickly despite known reliability issues.
• Lack of Algorithmic Transparency: Management keeps AI inner workings proprietary, preventing external scrutiny.

Technology

• Unreliable Speech Recognition: System phonetically transcribed German words as nonsensical English words.
• Invalid Intonation Analysis: Algorithm pulled personality traits from voice intonation, which is invalid.
• Visual Background Bias: Algorithm altered candidate scores based on video backgrounds and glasses.

Data Inputs

• Untested Language Inputs: Systems were not trained or tested on non-English inputs like German.
• Unstructured Open-Ended Data: Open-ended audio responses provide unstructured data hard for AI to evaluate.

Human Factors

• Overreliance on AI Scores: Hiring managers trust automated scores without verifying their validity.
• User Misunderstanding of AI: Users assume AI evaluates actual job skills rather than superficial cues.

Process and Methods

• Lack of Independent Testing: AI interviewing tools are not independently tested for accuracy or bias.
• Flawed Validation Methods: Companies use unstandardized methods to measure complex traits.

Regulatory Environment

• Lack of AI Hiring Regulations: Absence of legal standards governing AI use in employment decisions.
• No Oversight on Algorithmic Bias: Regulatory bodies do not audit automated hiring platforms for discrimination.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, first-hand experimental evidence of the AI systems' failures when tested with German inputs, along with expert commentary and responses from the AI developers.
• Ambiguities identified: The exact number of employers using these specific systems and the precise scale of job seekers affected are not quantified.

An investigation into commercial AI hiring platforms revealed significant reliability and bias issues, such as failing to detect non-English inputs and altering scores based on irrelevant visual cues. While highlighting critical gaps in AI robustness and potential civil rights concerns regarding employment discrimination, the incident poses negligible direct threats to national security.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This represents an ongoing, systemic challenge regarding AI bias and commercial software standards rather than an immediate crisis.
• Autonomy: Human-supervised. The AI systems automatically evaluate and score candidates, but final hiring decisions and tool deployment remain under human oversight.
• Novelty: Established threat. Algorithmic bias, lack of robustness to out-of-distribution inputs, and proxy-based evaluations are well-documented issues in AI systems.

Impact by dimension

• Physical security: Negligible. The incident involves commercial hiring software and has no bearing on physical systems, kinetic capabilities, or critical infrastructure.
• Information security: Negligible. No intelligence assets, classified models, or information warfare operations were involved in this software robustness failure.
• Sovereignty: Negligible. The incident does not impact state authority, electoral systems, or core government administrative decision-making.
• Economic security: Negligible. While highlighting commercial AI software limitations, the incident does not threaten national economic stability, critical supply chains, or strategic technological advantages.
• Societal stability: Minor. The use of unvalidated hiring tools can lead to systemic bias and unfair discrimination against job seekers, though the national security impact remains minor and manageable under existing labor regulations.