Personal voice assistants struggle with black voices, new study shows

A study found that voice recognition tools from Apple, Amazon, Google, IBM, and Microsoft disproportionately made errors when transcribing black speakers.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The commercial speech recognition systems exhibit significantly higher word error rates for Black speakers compared to white speakers, demonstrating unequal performance across demographic groups.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The downstream application of these biased transcriptions in job interviews or courtrooms could lead to active unfair discrimination against Black individuals.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The transcription disparities are directly generated by deployed AI speech recognition models, representing an unintentional bias stemming from their training and acoustic design.

EU AI Act risk tier

• Risk tier: 2 High Risk

Risk Level 2. High Risk: The report describes ASR systems being used in high-risk domains such as 'employment and worker management systems, such as AI used in recruitment' (evaluating candidate interviews) and law enforcement/justice (transcribing courtroom proceedings).

AI system and alleged parties

• AI system: Amazon ASR, Apple ASR, Google ASR, IBM ASR, Microsoft ASR (Amazon, Apple, Google, IBM, Microsoft)
• AI purpose: Voice Recognition
• Behaviour type: Tool
• Alleged developer: Microsoft, IBM, Google, Apple, Amazon
• Alleged deployer: Microsoft, IBM, Google, Apple, Amazon
• Alleged harmed parties: Black people

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Substantial
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment in the form of significantly higher speech recognition error rates for Black speakers compared to white speakers.

Directly caused: The five commercial ASR systems misidentified words 35 percent of the time for Black speakers compared to 19 percent for white speakers, with error rates reaching 41 percent for Black men.

Indirectly caused: The report notes that this performance gap makes it harder for Black Americans to benefit from voice assistants and could disadvantage them in professional and legal settings.

Inferred additional harm: Widespread deployment of these biased systems likely results in systemic differential treatment of millions of Black users attempting to use voice-activated technology daily.

Civil rights

Reported: The report explicitly describes potential civil rights implications regarding fair treatment in courtroom transcriptions and employment.

Indirectly caused: The report highlights that the disparity could actively harm Black Americans when speech recognition is used to automatically evaluate candidate interviews or transcribe courtroom proceedings, potentially violating civil rights to fair employment and due process.

Inferred additional harm: Inferred systemic violations of civil rights in legal and employment domains due to uncorrected transcription errors affecting legal outcomes and hiring decisions for Black individuals.

People affected

• Occurrences reported: 1
• People reportedly harmed: 73
• People reportedly exposed: 73

Potential causes

Management

• Lack of public performance reporting: Companies did not regularly assess or publicly report demographic accuracy.
• Underinvestment in inclusive data: Management failed to invest resources into collecting diverse voice datasets.

Technology

• Acoustic model performance gap: Acoustic models were confused by phonological and phonetic features of AAVE.
• Prosodic and pronunciation mismatch: Systems failed to handle rhythm, pitch, and vowel duration variations.

Data Inputs

• Unrepresentative training datasets: Training data lacked sufficient audio clips from black speakers.
• Insufficient AAVE audio data: Lack of AAVE speech data in training corpuses led to acoustic model failures.

Human Factors

• Dialect density variations: Higher dialect density in speakers correlated with higher system error rates.
• Gender-based speech differences: Black male speakers faced higher errors due to informal style and lenition.

Process and Methods

• Lack of demographic auditing: Developers failed to audit emerging ML systems for demographic inclusivity.
• Inadequate acoustic evaluation: Evaluation processes failed to detect acoustic performance gaps before release.

Regulatory Environment

• Absence of auditing standards: No regulatory requirements existed to mandate bias auditing in AI systems.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are based on a rigorous, peer-reviewed scientific study from Stanford published in PNAS, providing precise quantitative data, clear methodology, and thorough analysis of the underlying causes of the performance disparity.
• Ambiguities identified: The exact commercial models tested are proprietary, so their specific training datasets are not fully public, though the study infers the cause is lack of diverse acoustic training data.
• Alternative interpretations: One could argue the performance gap is partly regional rather than purely racial, though the study controls for this and demonstrates a strong correlation with African American Vernacular English (AAVE) dialect density.

A Stanford study found significant racial bias in commercial automated speech recognition systems from five major tech companies, with error rates for Black speakers nearly double those of white speakers. While this poses minor risks to societal stability and civil rights, particularly if used in courtrooms or job interviews, it does not present a direct threat to national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This represents an ongoing, systemic technological bias issue rather than an immediate national security crisis requiring urgent response.
• Autonomy: Human-controlled. The ASR systems act as tools to transcribe audio, with humans retaining control over final decisions in professional or legal environments.
• Novelty: Established threat. Demographic bias and training data disparities in machine learning models are well-documented and established issues in the AI field.

Impact by dimension

• Physical security: Negligible. The incident involves commercial speech recognition software and does not pose any threat to physical systems, critical infrastructure, or human safety.
• Information security: Negligible. No evidence of information warfare, intelligence compromise, deepfake operations by adversaries, or classified model theft is present in the report.
• Sovereignty: Minor. Potential downstream use of biased ASR systems in courtroom transcriptions could affect judicial processes, representing a minor risk to local government and legal functions.
• Economic security: Negligible. While there are potential individual economic impacts regarding job interviews, there is no threat to national economic stability, strategic industries, or tech competitive advantage.
• Societal stability: Minor. The study reveals systemic racial bias in major commercial AI systems, which could lead to unequal treatment and civil rights concerns for Black Americans in professional and legal settings.