Benchmark · frontier-ASR baseline

How a frontier ASR hears Bantu-accented English.

Every code-switch take is time-segmented at the Bantu→English boundary, so the English half is isolated, transcribed by a frontier ASR (Google Cloud Speech-to-Text), and scored against the gold reference. The result is a measured, reference-backed accent-robustness benchmark — run your own model on the same audio and compare to this baseline.

Evaluate it See the corpus

227Takes scored

2Languages

81%Content recall

69%“means” misheard

The systematic failure mode

The accent breaks function words before it breaks meaning.

The substantive English is mostly recovered (content recall 81%), but the discourse marker means is mis-transcribed in 69% of takes — heard as “men's”, “main”, even “Main Street”. A high-frequency word a frontier ASR gets wrong almost every time on this accent. That is exactly the gap an accent benchmark is built to surface.

🇸🇿 Swati215 🇿🇲 Bemba12

Listen + compare

Gold reference vs the ASR hypothesis.

Play the English segment (what the STT was given), then read the gold reference against Cloud STT's transcript.

akanya nga fwaya ukuya kucimbushu “means” misheard

akanya nga fwaya ukuya kucimbushu

means she will defecate when she feels like going to the toilet

Audio pending.

Gold refmeans she will defecate when she feels like going to the toilet

Cloud STTmen's shoes defecate when she feels like going to the toilet

confidence 0.84 content recall 89% WER 0.30

akanya nga fwaya ukuya kucimbushu not detected as English

akanya nga fwaya ukuya kucimbushu

means he will defecate when he feels like going to the toilet

Audio pending.

Gold refmeans he will defecate when he feels like going to the toilet

Cloud STTyou defecate

confidence 0.48 content recall 11% WER 0.90

Akabomba ku ofesi namumushi nga afika not detected as English

Akabomba ku ofesi namumushi nga afika

means he will work at the office tomorrow when he arrives

Audio pending.

Gold refmeans he will work at the office tomorrow when he arrives

Cloud STTmemes

confidence 0.45 content recall 0% WER 1.00

Alelanda ne mfumu nga bafika ku ng'anda not detected as English

Alelanda ne mfumu nga bafika ku ng'anda

means he was speaking with the chief when they arrived at the house

Audio pending.

Gold refmeans he was speaking with the chief when they arrived at the house

Cloud STTmeans

confidence 0.59 content recall 0% WER 1.00

Abantu babomba ku ofesi lyonse not detected as English

Abantu babomba ku ofesi lyonse

means the people work at the office every day

Audio pending.

Gold refmeans the people work at the office every day

Cloud STTmeans

confidence 0.55 content recall 0% WER 1.00

Bafwile mu nshita ya nsala ya myaka iyibili content review

Bafwile mu nshita ya nsala ya myaka iyibili

means they died in the famine of two years ago

Audio pending.

Gold refmeans they died in the famine of two years ago

Cloud STTmeans

confidence 0.64 content recall 0% WER 1.00

akanya umunani content review

akanya umunani

means share the relish

Audio pending.

Gold refmeans share the relish

Cloud STTmeans share the radish

confidence 0.66 content recall 50% WER 0.50

ulya akanya! content review

ulya akanya!

means that one will face severe consequences

Audio pending.

Gold refmeans that one will face severe consequences

Cloud STTmeans that one who said severe consequences

confidence 0.60 content recall 67% WER 0.33

akanya kalelila content review

akanya kalelila

means the baby is crying

Audio pending.

Gold refmeans the baby is crying

Cloud STTmeans the babies crying

confidence 0.74 content recall 33% WER 0.67

Per-language intelligibility

Measured accent difficulty.

Language	Scored	Content recall	Deviation WER	“means” err	Not-English	Mean conf.
🇸🇿 Swati	215	82%	1.14	72%	7%	0.73
🇿🇲 Bemba	12	46%	0.56	25%	33%	0.66

The offer

A turnkey accent-robustness eval, with a frontier baseline.

Run your ASR on the same time-segmented audio, score against the gold references, and compare to this Cloud-STT baseline. It is verification/validation test data for Bantu-English speech — native, consented, reference-backed, and growing. Pull it from /api/v1/tone/asr-benchmark.

Start your free evaluation

How a frontier ASR hears Bantu-accented English.

The accent breaks function words before it breaks meaning.

Gold reference vs the ASR hypothesis.

Measured accent difficulty.

A turnkey accent-robustness eval, with a frontier baseline.

Start where you are.