Benchmark · frontier-ASR baseline
How a frontier ASR hears Bantu-accented English.
Every code-switch take is time-segmented at the Bantu→English boundary, so the
English half is isolated, transcribed by a frontier ASR (Google Cloud Speech-to-Text),
and scored against the gold reference. The result is a measured, reference-backed
accent-robustness benchmark — run your own model on the same audio and
compare to this baseline.
The systematic failure mode
The accent breaks function words before it breaks meaning.
The substantive English is mostly recovered (content recall 81%),
but the discourse marker means is mis-transcribed in
69% of takes — heard as “men's”, “main”, even
“Main Street”. A high-frequency word a frontier ASR gets wrong almost every time on this
accent. That is exactly the gap an accent benchmark is built to surface.
Listen + compare
Gold reference vs the ASR hypothesis.
Play the English segment (what the STT was given), then read the gold reference against Cloud STT's transcript.
Kulula tinyawo kudzingekile mawuvuka ekuseni.
“means” misheard
Kulula tinyawo kudzingekile mawuvuka ekuseni.
means stretching ones feet is necessary when waking up in the morning
Audio pending.
Gold refmeans stretching ones feet is necessary when waking up in the morning
Cloud STTmen's stretching one's feet is necessary when waking up in the morning
confidence 0.88
content recall 89%
WER 0.44
Speaker SSW-A
Libala njengenkhukhu, utotibonela.
“means” misheard
Libala njengenkhukhu, utotibonela.
means be stupid like a chicken, you will suffer the consequences
Audio pending.
Gold refmeans be stupid like a chicken, you will suffer the consequences
Cloud STTmen's be stupid like a chicken you'll suffer the consequences
confidence 0.83
content recall 88%
WER 0.38
Speaker SSW-A
Kutsela umtselo wenhloko akusadzingeki.
“means” misheard
Kutsela umtselo wenhloko akusadzingeki.
means paying graded tax is no longer necessary
Audio pending.
Gold refmeans paying graded tax is no longer necessary
Cloud STTmens paying credit tax is no longer necessary
confidence 0.78
content recall 86%
WER 0.29
Speaker SSW-A
Libala lakhe linjengelala bamhlophe.
“means” misheard
Libala lakhe linjengelala bamhlophe.
means his complexion is like that of a white person
Audio pending.
Gold refmeans his complexion is like that of a white person
Cloud STTmen's his complexion is like that of a white person
confidence 0.90
content recall 100%
WER 0.29
Speaker SSW-A
Lapha lomkuhlane ungakakubulali.
“means” misheard
Lapha lomkuhlane ungakakubulali.
means treat the flu before it kills you
Audio pending.
Gold refmeans treat the flu before it kills you
Cloud STTMain Street the flu before it kills you
confidence 0.80
content recall 80%
WER 0.40
Speaker SSW-A
Lapha kulapho sisukela khona.
“means” misheard
Lapha kulapho sisukela khona.
means here is where we start from
Audio pending.
Gold refmeans here is where we start from
Cloud STTmen's hair is where we start from
confidence 0.76
content recall 100%
WER 0.50
Speaker SSW-A
Phatsa onkhe emaswidi lakhona.
“means” misheard
Phatsa onkhe emaswidi lakhona.
means carry all available sweets
Audio pending.
Gold refmeans carry all available sweets
Cloud STTmen's carry all available Suites
confidence 0.75
content recall 100%
WER 0.75
Speaker SSW-A
Babetala
“means” misheard
Babetala
means father in law
Audio pending.
Gold refmeans father in law
Cloud STTmen's father-in-law
confidence 0.54
content recall 100%
WER 1.00
Speaker SSW-A
Yetsa lemoto ingakacishi.
“means” misheard
Yetsa lemoto ingakacishi.
means fill the car before it turns off
Audio pending.
Gold refmeans fill the car before it turns off
Cloud STTMansfield the car before it turns off
confidence 0.79
content recall 80%
WER 0.20
Speaker SSW-A