Tarazu ran a bake-off of four speech-to-text (STT) vendors on 16 kHz app recordings, and vendor A came first. On Jio-to-trunk audio, ranked by amount entity error rate, vendor A came third of four.
What happened in the bake-off?
The bake-off measured the vendors on audio Asha never hears. Asha receives 8 kHz G.711 mu-law in 20 ms frames, often after a mobile codec and at least one transcode, and the ranking on that audio was different. The vendor that would have shipped was the one that degraded most on the phone line.
Here is the path in a trace (T01, where "dhai hazaar" became "do hazaar"):
bake-off 4 STT vendors, same lines, scored by amount entity error rate
16 kHz app recordings vendor A ranked 1 of 4
Jio -> trunk, 8 kHz vendor A ranked 3 of 4
trace TZ-2026-11-14-0871234 (T01)
"carrier": "jio",
"codec_chain": ["amr-nb", "g711-pcma", "g711-pcmu"],
"audio": {"sample_rate_hz": 8000, "carrier_offset_ms": 212}
Diagnosis: the evaluation audio did not match the deployment channel. Three codecs sit between the borrower's mouth and the STT on T01: AMR-NB on the mobile leg, A-law on part of the trunk, and mu-law into the media stream. None of them was in the bake-off, so the ranking measured the wrong channel. The trace fields are defined in what a voice agent trace contains.
What does a phone line remove?
A narrowband phone channel samples at 8 kHz and passes roughly 300-3400 Hz, so nothing above 4 kHz survives and the lowest band goes too. A model trained only on 16 kHz audio gets input unlike anything it learned from, and its error rate can roughly double on the same speech.
Narrowband audio is the name for this channel. At 8,000 samples a second the highest representable frequency is 4 kHz, and the telephone band filter cuts below that, at about 3.4 kHz. Fricatives such as "s", "sh" and "f" carry much of their energy above 4 kHz, so on a call they become harder to tell apart, and short words that differ by one consonant have less left to separate them.
The measured cost is large. In a Microsoft voice-search study, a deep neural network trained only on wideband audio went from 27.5% WER on 16 kHz test audio to 53.5% on the same audio downsampled to 8 kHz, and adding 8 kHz data to training brought it down to 29.3% 1.
Figure: A schematic, not a measurement. Synthetic Tarazu data. The shaded band is what a call carries; the grey regions are gone before the STT sees a byte, and no model downstream can bring them back. In the cookbook pipeline's synthetic run, the share of energy above 3.4 kHz fell by 65.7 dB through the band-pass and the move to 8 kHz.
Upsampling an 8 kHz call to 16 kHz because a model expects 16 kHz input restores nothing; the model gets a 16 kHz file with an empty top half. Test a vendor's telephony model and its general model on your phone audio, because the names do not tell you which one wins.
Is mu-law the problem?
Mostly not. G.711 mu-law companding adds little audible damage on its own; the loss comes from the 300-3400 Hz band and from transcoding between codecs. Judge voices and models on 8 kHz G.711 audio, never on a wideband demo.
Mu-law is the G.711 companding curve: it spends more of its 256 code values on quiet samples, so a quiet borrower and a loud one get similar coding accuracy. ETSI TR 102 948 rates G.711 as natural sounding, with no perceptual degradation (an informal expert judgement) 2. The cookbook pipeline shows why with its own synthetic run: on a phone-band signal at -3 dBFS peak, mu-law gives 37.7 dB round-trip SNR against 40.9 dB for plain 8-bit linear coding; at -33 dBFS, mu-law still gives 34.8 dB while linear falls to 9.5 dB. Mu-law moved 2.9 dB over a 30 dB drop in level.
So a 24 kHz TTS voice that sounded best in the demo can lose its lead on a call because it loses the high-frequency detail that made the demo sound good. The Blizzard Challenge 2009 tested this the hard way: over a deliberately low-quality simulated phone channel (G.712 bandpass plus a 16 kbit/s G.726 tandem), no synthetic voice matched natural speech for intelligibility, and the organisers suggested synthetic speech may suffer more from the channel than natural speech 3. Tandems are the part teams forget: T01 crossed three codecs, and each decode and re-encode adds its own error.
The framing matters as much as the codec. Twilio Media Streams always delivers audio/x-mulaw at 8000 Hz, mono, and requires audio sent back to be mu-law at 8000 Hz, base64 encoded 4. Tata Tele Business Services' Smartflo bidirectional streaming uses the same format and asks that audio sent to it be 160 bytes or a multiple of 160 bytes, warning that otherwise "audio gaps might occur" 5. At 8,000 one-byte samples a second, 160 bytes is 20 ms, which is also RTP's default packetisation interval for G.711 6. Asha's TTS requests 8 kHz mu-law output directly, and the media-out stage sends 160-byte frames. If your code resamples TTS audio itself, a chunk that is not a whole 20 ms frame can create a gap on the call that a transcript-only test never hears.
What does packet loss do to STT?
Packet loss is serious, but no study found ranks it as the single worst. In a 2022 Frontiers study, STT error rose once loss passed 10% (clean-trained model) or 15% (distortion-trained), and at 0 dB SNR noise dominated loss. Larger models tolerate loss better; WER above 100% means the model inserted words nobody said.
On a call a lost packet is 20 ms of missing speech, and loss tends to come in bursts: a radio handover or a congested link drops several packets in a row.
The evidence, stated with its scope:
- Thresholds. In a 2022 Frontiers study (Kumalija and Nakamoto; G.722, no jitter buffer or packet loss concealment), WER rose once loss passed 10% for a clean-trained model and 15% for a model trained on distorted speech, and at 0 dB SNR noise dominated packet loss 7. Read those thresholds as an order of magnitude for G.711.
- Human listeners. In a 2021 ACM TACCESS study of listeners with hearing loss, packet loss affected speech recognition, mental effort and quality ratings, while bit rate affected speech recognition for narrowband audio and quality ratings for both narrowband and wideband 8. That is the agent-to-borrower direction, and it describes callers like P13, Shankar Rao, 64 and hard of hearing.
- Model size. Dissen et al. (Technion, Interspeech 2024) simulated loss by zeroing mel-spectrogram frames. Whisper large-v2 was "only starting to seriously degrade" above about 20% frame loss (French WER 7.2 at 0%, 10.3 at 20%, 27.5 at 40%), while Whisper base degraded at once (24.7 at 0%, 48.6 at 20%, 104.9 at 40%) 9. Zeroed spectrogram frames are not RTP packet loss with a codec's own concealment, so the result transfers only in direction.
WER can pass 100%, as in that 104.9, only through insertions: the model wrote words nobody said. The paper does not call it hallucination; that is this page's reading. For a collections agent an inserted number is an amount nobody said, so under heavy loss count hallucinated entities, as defined in entity error rate, first.
Model loss as bursts on 20 ms frames, at several rates, and report STT accuracy against the loss each clip actually received.
Does packet loss concealment help STT?
Not reliably. Packet loss concealment (PLC) fills lost frames so a listener hears better audio, but in the ICASSP 2024 PLC challenge the best-sounding systems raised word accuracy by 0.01 at most, and one cut it sharply. Judge any concealment that feeds STT by word accuracy on your own audio.
PLC is the receiver step that invents audio for frames that never arrived: repeat the last frame, fade it, or generate a continuation with a model. It is tuned for people, who forgive a plausible guess.
The ICASSP 2024 Audio Deep Packet Loss Concealment Grand Challenge (Microsoft; 600 real loss traces, full-band audio, ITU-T P.804 crowd ratings plus word accuracy) separates the two goals. The two co-winners raised overall quality from 2.37 for the unconcealed lossy audio to 3.49 and 3.44, but their word accuracy was 0.81 and 0.84 against 0.83 for the lossy input, and the eighth-placed system cut word accuracy to 0.64 10. A concealer that sounds better can make recognition worse, because a plausible invented frame is still invented, and the STT has no way to know which 20 ms are real.
Two decisions follow. PLC for the borrower's ear and PLC in front of your STT are separate settings where your media stack allows it. And a vendor's claim that its stack "handles packet loss" says nothing about your amount entity error rate until you run your loss traces through it.
Why do codec effects differ by language and corpus?
Because the damage depends on the cues a language and speaking style rely on, and on how each model was trained. In one 2026 study, G.711 cost English conversational audio under one WER point on average and Japanese far more; on English read speech one model lost 42.5 points while six lost under one.
WildASR (Tay et al., Boson AI, March 2026) downsampled audio to 8 kHz, passed it through G.711 mu-law or GSM, and resampled it to 16 kHz. Across seven models, the average error rose +0.6 WER on English conversational audio (MagicData) but +6.3 on English read speech (FLEURS), +9.4 CER on Japanese conversational, and up to +28.0 CER for GSM on Korean conversational. The English FLEURS G.711 average is driven by one model: Qwen2-Audio lost 42.5 WER points, while the other six lost 0.6 or less 11.
Read three things out of that:
- The penalty belongs to the pair (language, model), not to the codec. An English number says nothing reliable about Hinglish.
- An average across models hides the model that breaks. Report each vendor per condition, never a pooled "telephony penalty".
- The study does not cover your path. It has no Hindi or Hinglish, no AMR or Opus and no packet loss; T01's chain is in no published table this page found.
The only defensible number for Tarazu's path is one measured on it, sliced by language mix. Code-switch density is a separate slice, owned by normalization and code-switching in Hinglish STT.
How do you build the phone path offline?
Take consented 16 kHz recordings, add noise at a fixed SNR, band-pass to 300-3400 Hz, resample to 8 kHz, run the codec chain (AMR-NB, A-law, mu-law), drop 20 ms frames with a burst model, and feed the STT in its production streaming mode. Use the ITU-T Software Tool Library for numbers you will defend.
flowchart LR
A["Consented 16 kHz clip"] --> B["Noise at fixed SNR<br/>(at the handset)"]
B --> C["Band-pass 300-3400 Hz"]
C --> D["Resample to 8 kHz"]
D --> E["AMR-NB encode and decode"]
E --> F["Transcode A-law to mu-law"]
F --> G["Burst loss on 20 ms frames<br/>Gilbert-Elliott"]
G --> H["STT, production streaming<br/>and code-mix settings"]
H --> I["Canonicalize and score<br/>entity error rate"]
The order is part of the model. Background noise reaches the handset microphone before any codec, so the codec encodes noise and speech together; noise added after decoding is cleaner than any borrower's kitchen. Loss happens on the network, after encoding, and removes whole coded frames.
The burst model is a two-state Gilbert-Elliott chain on 20 ms frames: a good state (frames arrive) and a bad state (frames are lost). With p the chance of entering the bad state and r the chance of leaving it, the long-run loss rate is p/(p+r) and the mean burst is 1/r frames. Fix r = 0.5 for 40 ms mean bursts and solve for p: a 1% target needs p = 0.0051, 5% needs 0.0263, and 10% needs 0.0556.
| Degradation | What it simulates | Parameters | Which metric moves | Source |
|---|
| Band-pass and 8 kHz resample | The telephone channel | 300-3400 Hz, 8,000 samples a second | STT error on fricatives and close number words; TTS clarity | Yu et al. 2013 |
| G.711 mu-law or A-law | Trunk and media stream coding | mu = 255; 160-byte, 20 ms frames | Little on its own; check frame alignment | ETSI TR 102 948 |
| Codec tandem | Mobile leg plus trunk transcodes | AMR-NB -> A-law -> mu-law (T01's chain) | STT error, unknown size per language | No published table for this chain |
| Burst packet loss | Radio handover, congested links | Gilbert-Elliott, 1%, 5%, 10%, mean burst 40 ms | Deletions, then insertions past heavy loss | Kumalija and Nakamoto 2022; Dissen et al. 2024 |
| Concealment | The receiver's PLC | Zero fill, repeat, model-based | Word accuracy can fall while quality rises | ICASSP 2024 PLC challenge |
| Noise at fixed SNR | Kitchen, TV, street | Clean, 20 dB, 10 dB | STT error; dominates loss at 0 dB SNR | Kumalija and Nakamoto 2022 |
The teaching version is a few hundred lines of numpy in the phone-path degradation recipe. For numbers you will defend in a vendor review, use reference code. ITU-T G.711 is still the 11/1988 text, with Appendix I (09/1999) giving a standard concealment algorithm and Appendix II (02/2000) covering comfort noise 12. Reference C code for G.711, the Appendix I concealment (g711iplc), the error insertion device (eid), MNRU, the sv56 level meter and the FIR and IIR filters is in the ITU-T Software Tool Library (G.191), open on GitHub, latest release v2024 (2 September 2024). It is a more defensible phone-path simulator than ad hoc code 13. The STL does not include AMR-NB, so that leg needs a separately licensed or reference build; until you have one, mark the AMR rows of the matrix as not run rather than approximating them.
What does one amount utterance look like through five conditions?
The cookbook pipeline turns a 4-second clip into five files: the 16 kHz source, the plain phone path, plus 5% burst loss, plus noise at 10 dB SNR, and both. Its checks show the band removed, 200 whole 20 ms frames, and a loss draw of 20 frames with one 180 ms burst, twice the 5% target.
The recipe's run prints (synthetic Tarazu data, seed 20261120):
| File | Condition | What the run shows |
|---|
| 00_source_16k.wav | 16 kHz source | Energy above 3.4 kHz at -2.9 dB of the total |
| 01_phone_mulaw.wav | Band-pass, 8 kHz, mu-law | That share falls to -46.0 dB after mu-law; 32,000 bytes = 200 frames of 160 bytes |
| 02_phone_mulaw_loss5.wav | Plus 5% burst loss | 20 of 200 frames lost (10%), bursts of 9, 1, 1, 4, 1, 1, 2 and 1 frames |
| 03_phone_mulaw_snr10.wav | Plus noise at 10 dB | Realised SNR 10.00 dB |
| 04_phone_mulaw_snr10_loss5.wav | Both | Same loss mask, same noise |
This page prints no transcripts: the course has no approved speech clip yet, and a chirp has no words. Pass your own consented recording of an amount line ("dhai hazaar rupaye bees tarikh tak") through the same script and fill two columns per file: the STT output and whether the canonical amount survives.
The loss row is the one to study before you have any transcripts. A 9-frame burst is 180 ms of silence. At a typical speaking rate of 4.4 syllables a second (the TW campaign's median agent rate in T10; the course has no measured rate for borrowers), one syllable lasts about 227 ms, so that burst can erase most of a one-syllable word, and "dhai" is one syllable. Where it lands decides whether the clip loses a filler or the amount, so log the realised loss mask with every file and run many clips per condition.
How do you test the live path?
Impair a real SIP test call with Linux netem: burst loss, delay and jitter on the media path. Offline frame dropping tests the recogniser; a live call also exercises the jitter buffer, concealment, voice activity detection and barge-in. Run both before release.
Linux netem can impose loss, delay and jitter on a live test call's network path, including Gilbert-Elliott burst loss with the syntax gemodel PERCENT [ R [ 1-H [ 1-K ]]] (the chance of entering the bad state, of leaving it, and the loss probabilities in the bad and good states) and delay with jitter drawn from uniform, normal, pareto or paretonormal distributions 14. The 5% target from the offline model maps directly:
# on the test box interface that carries the SIP media
sudo tc qdisc add dev eth0 root netem loss gemodel 2.63% 50% 100% 0%
sudo tc qdisc change dev eth0 root netem delay 40ms 20ms distribution normal loss gemodel 2.63% 50% 100% 0%
sudo tc qdisc del dev eth0 root netem
What the live call finds that the offline files cannot:
- The jitter buffer. A buffer that grows under jitter adds delay to every turn gap; measuring voice agent latency owns that number.
- The production concealment. Your media stack's PLC runs, not the one you modelled.
- Voice activity detection. A lost burst looks like a pause, so the turn detector can commit on it. Count early jump-ins under loss; turn-taking, barge-in and recovery defines them.
- Barge-in. The barge-in monitor hears the caller through the same impaired path. Stop latency under jitter is its own number.
Offline files tell you, cheaply and on thousands of clips, whether the STT copes with missing audio. Live calls tell you, slowly and on tens of calls, whether the rest of the stack copes. Neither replaces the other.
Which audio quality metrics apply to which direction?
Split by direction. Agent-to-borrower audio is judged by what a listener hears, so listening metrics apply; borrower-to-STT audio is judged by what the recogniser gets right, so task metrics such as entity error rate and word accuracy apply. POLQA is licensed, PESQ is withdrawn, and STOI barely sees narrowband damage.
The mistake to avoid is scoring the inbound direction with a listening metric: the PLC result above shows listener scores and word accuracy moving in opposite directions.
| Direction | Question | Metrics that fit | Owner |
|---|
| Borrower -> STT | Did the recogniser get the words and entities right? | Amount and date entity error rate, hallucinated entities, WER as a tie-breaker | Entity error rate |
| Asha -> borrower | Can the borrower hear and understand it? | Listening MOS predictors, panels, spoken-entity round trip on the phone path | Evaluating speech output for agents |
The intrusive listening metrics, which compare a degraded file against its clean reference, need care:
- POLQA (ITU-T P.863). The in-force edition is still 03/2018 (reposted 12 October 2021), with Amendment 1 (04/2020) for acoustically recorded narrowband speech 15. POLQA is a patented, licensed algorithm, and OPTICOM is the sole licensing agent 16, so an open lab cannot ship it. Like PESQ it predicts a listening MOS, not recognition accuracy, so it belongs on the agent-to-caller direction.
- PESQ (ITU-T P.862). ITU-T lists every part of the P.862 family as withdrawn or superseded; its catalogue page says P.862, P.862.1, P.862.2 and P.862.3 were deleted on 5 January 2024 and points to P.863 17. It is still everywhere: Torcoli, Halimeh and Habets count over 4,600 Google Scholar results for PESQ in 2024 against a little more than 100 for POLQA, and show that PESQ versions disagree by an RMSE of 0.56 to 0.62 against the ITU wideband reference without Corrigendum 2 18. A PESQ number is only comparable when the version, implementation and sample rate are stated.
- STOI. The reference Python STOI (pystoi 0.4.1) resamples both signals to 10 kHz and analyses 15 one-third-octave bands starting at 150 Hz, so its top band ends near 4.3 kHz and the 300-3400 Hz phone band removes only a few of its lowest and highest bands; it also refuses signals of unequal length 19. STOI is a weak detector of narrowband damage, needs time-aligned simulation output, and is not a stand-in for word error rate.
Non-intrusive MOS predictors and listening panels belong to the TTS chapter; whatever you use must run on phone-path audio.
Which conditions belong in the Tarazu matrix?
Five factors: carrier, codec chain, loss, SNR and language mix, at 4 x 3 x 3 x 3 x 3 = 324 cells. Tarazu runs 36: every carrier x codec chain x loss combination once, with SNR and language mix assigned so every pair of levels across every two factors appears at least once. A script checks all 102 pairs.
The factors and levels:
| Factor | Levels |
|---|
| Carrier | 4 carrier profiles |
| Codec chain | mu-law; A-law -> mu-law; AMR-NB -> mu-law |
| Packet loss | 1%, 5%, 10% (Gilbert-Elliott, 40 ms mean bursts) |
| SNR | Clean, 20 dB, 10 dB |
| Language mix | Hinglish, Hindi, Indian English |
A carrier profile is worth a level only if it differs from the others. Build each from live test calls on that carrier: the mobile codec mode it negotiates, its measured loss and jitter, and its level.
The 36-row design runs each of the 36 carrier x codec chain x loss combinations once, and sets SNR to level (codec + loss) mod 3 and language mix to level (codec + 2 x loss + carrier) mod 3. There are 10 factor pairs: the 4 that involve the carrier have 4 x 3 = 12 level pairs each (48), and the 6 among the four three-level factors have 9 each (54), so 102 level pairs in all. A brute-force check over the 36 rows finds every one of them. Each carrier appears in 9 rows and each level of every other factor in 12.
What the covering design buys and what it does not:
- It catches pairwise interactions. If AMR-NB fails only on Hindi, some row has both. A one-factor-at-a-time plan would not contain that pair.
- It does not give a precise estimate per cell. Gate on factor levels (all 12 rows with 10% loss, all 9 rows for carrier 3), where the data pools, and use cells to locate a failure, not to certify one.
- Pair the clips. Language mix is a property of the utterance, so each row draws clips in its language; within a language the same clips go through every row, which makes condition comparisons paired.
- Three-way interactions are not covered. AMR-NB x 10% loss x 10 dB SNR may never meet. The real holdout exists partly for that.
For two vendors on the same clips, the statistic is the paired difference in amount entity error rate, bootstrapped by speaker; bootstrap and paired tests has the method.
What gate would have caught it?
A rule that STT and TTS gates run only on phone-path audio. The bake-off would have scored vendors on the 36-row matrix and on a real 8 kHz holdout from consented calls, with the 16 kHz numbers in a separate column that never gates. Vendor A would have ranked third before the shortlist, not after.
The gate:
gate: stt-phone-path
audio:
simulated: consented 16 kHz volunteer clips through phone-path v1 (36-row matrix)
real_holdout: gold-v7 STT set (1,200 reference-transcribed caller utterances, 180 speakers, 8 kHz, stratified by carrier)
wideband: reported in a separate column; never gates
metric: amount and date entity error rate per language group, hallucinated entities alongside
interval: speaker-clustered bootstrap, 95%
block_if:
- real_holdout upper_bound > campaign_threshold
- any factor level (pooled rows) upper_bound > campaign_threshold
- paired_difference_upper_bound > margin # candidate minus incumbent, same clips, by speaker
report_always: [per-cell results, realised loss per clip, rows not run (AMR without a reference build)]
The interval rule: utterances that share speakers are clustered, so use a speaker-clustered bootstrap, not Wilson, and a paired bootstrap by speaker for a vendor difference. Thresholds and margin are the campaign owner's numbers, as in entity error rate.
Why both audio sources: the simulator covers only the impairments you thought to model. The real holdout catches the rest (a handset's noise suppression, a transcode you did not know about) and is the ground truth for the simulator. If the matrix and the holdout rank vendors differently, trust the holdout and fix the simulator.
The same rule holds for TTS: the spoken-entity round trip in evaluating speech output for agents runs on 8 kHz mu-law.
What goes in the decision memo?
The memo fixes phone path v1: which degradations, which levels, what runs offline and what runs live, which holdout, and the rule that wideband never gates. Tarazu's worked version follows.
Decision. Adopt phone path v1 for every STT and TTS gate: the 36-row matrix offline, netem on SIP test calls before release, and the gold-v7 STT set as the real 8 kHz holdout. Wideband results are reported, never gated.
Context. The 16 kHz bake-off ranked vendor A first; on Jio-to-trunk audio by amount entity error rate it ranked third of four. Production audio is 8 kHz G.711 mu-law in 20 ms frames, often after AMR-NB and an A-law transcode (T01).
Options considered. (1) Keep the 16 kHz bake-off: cheap, and it picks the wrong vendor. (2) Real calls only: true audio but no control over conditions and slow to grow. (3) Simulated matrix plus a real holdout plus live netem calls: chosen.
Evidence. Wideband-only training nearly doubled WER at 8 kHz in Yu et al. 2013; codec penalties varied by language and model in WildASR (2026); PLC raised quality without raising word accuracy by more than 0.01 (ICASSP 2024). Gates use speaker-clustered 95% intervals on gold-v7.
Owner and cadence. Eval platform team owns the path; the STT owner runs it on every vendor model version and quarterly; carrier profiles refresh quarterly from live test calls.
Tripwires. Matrix and holdout rankings disagree; a new carrier or codec in call traces; AMR rows still not run at the next release.
Dated facts. WildASR v1 (March 2026); ITU-T P.863 page (April 2026); STL v2024.
What to do this week
- List every
codec_chain and carrier in a week of traces.
- Rerun your last STT comparison on 8 kHz mu-law audio, with whole 20 ms frames, and put the 16 kHz column next to it.
- Add burst loss at 1%, 5% and 10% with 40 ms mean bursts, log the realised loss per clip, and count hallucinated entities under the heaviest condition.
- Run one SIP test call through netem and listen for gaps, early cutoffs and missed barge-ins.
- Write the no-wideband-gating rule into your release checklist.
For the general method behind perturbation suites, see robustness testing; metric targets for voice live in the voice playbook.
NOTE
What will date on this page. WildASR is arXiv 2603.25727 v1 (26 March 2026); recheck for a version that adds Hindi or AMR. The ITU-T P.863 page showed the 03/2018 edition in force when last updated on 24 April 2026; the P.862 family was deleted on 5 January 2024. The ITU-T STL's latest release is v2024 (2 September 2024). The tc-netem page was read on 20 November 2026; pystoi is 0.4.1 (29 December 2023). Twilio's and Smartflo's stream formats were read on 20 November 2026 and can change without notice. Recipe numbers are tied to seed 20261120.