Picture two job candidates giving the same answer to the same interview question, word for word. One has an American accent. The other has a Nigerian accent. Ask a leading AI model to score them, and the American will likely come out ahead.
That’s the gist of new research from the University of Washington, which tested how AI models from Google, OpenAI and Alibaba evaluated English spoken with five accents: American, British, Chinese, Indian and Nigerian.
Across job interviews, conference presentations and English proficiency tests, the models generally ranked American and British speakers highest, Indian speakers in the middle, and Chinese and Nigerian speakers lowest.
The penalty fell mostly on delivery. But content scores for Chinese and Nigerian speakers sometimes slipped, too, even though every speaker read the same script and the models were told to judge only the substance.
“Even though they repeatedly claimed Chinese and Nigerian speakers ‘mumbled’ or were ‘difficult to understand,’ the models were actually very good at transcribing the accented speech,” said Orevaoghene Ahia, a UW doctoral student in the Paul G. Allen School of Computer Science & Engineering and co-lead author of the study.
The stakes go beyond a bad review. Companies increasingly use AI to screen recorded job interviews, and instructors are turning to oral exams as written work gets easier to fake with AI. At least one major English test used for visas and immigration, Pearson’s PTE Academic, relies on AI to score some spoken answers. It was given about 1.2 million times in 2023, according to figures cited in the paper.
The researchers didn’t test the actual products used by hiring platforms or test makers. How those systems work is usually not public, and the paper notes there are no public audits of AI-scored English tests that the authors know of. Instead, the team tested general-purpose models that companies could build on.
“We wanted to check these state-of-the-art models because these are what companies would deploy in a real-world setting,” said Yulia Tsvetkov, a UW associate professor in the Allen School and the study’s senior author.
Listen to the five accents below.
American:
Nigerian:
Chinese:
Indian:
British:
The research was funded in part by gifts from Google and OpenAI, whose models were among those tested.
To understand the pattern, the researchers measured how acoustically different each accent was from American English. Chinese and Nigerian accents were the farthest away, and models’ scores dropped as that distance grew. Indian and British accents sat close to the American baseline.
The researchers suggest the bias may come from training data that describes accented speakers as unclear, from data sets that underrepresent some accents, or from training that reinforces ideas about how a professional should sound. English is an official language in Nigeria and the language of instruction in Indian schools, yet both accents scored below American and British ones.
The study is the latest in a run of UW research on AI bias. Earlier projects found that AI models favored resumes with white- and male-associated names, that politically slanted chatbots could shift people’s views, and that AI-written children’s stories nearly erased female animal characters.
The team presented the research Oct. 25 at the Conference on Empirical Methods in Natural Language Processing in Budapest. Next, the researchers plan to study more accents, including regional variations within countries, and to look at how AI models handle audio in medical settings such as automated emergency response.
“For all of these, we hope that developers build and publish evaluations of their models that don’t focus only on average performance, but also break it down into subpopulations, dialects and so on,” Tsvetkov said. “We don’t want the average performance to be 97% accurate for the general population, but for a smaller group only 20%.”
