Steelyard Credit tested four speech-to-text (STT) vendors for Hazel, its AI collections agent, on 16 kHz recordings from its mobile app, and vendor A won. Scored by amount entity error rate on mobile-to-trunk audio (AMR-WB to G.711), the same vendor finished third of four.
What happened in the bake-off?
The vendors were ranked on audio that never reaches Hazel. Every call arrives as 8 kHz G.711 mu-law in 20 ms frames, usually after a mobile or VoIP codec and a transcode, and on that audio the order changed. The winner of the wideband test was the vendor that lost the most on the phone line.
The path shows up in the trace of T01, the call where Ryan Mitchell said "two fifty" and the STT wrote "two fifteen":
bake-off 4 STT vendors, same lines, scored by amount entity error rate
16 kHz app recordings vendor A ranked 1 of 4
mobile -> trunk (AMR-WB to G.711) vendor A ranked 3 of 4
trace SY-2026-11-14-0871234 (T01)
"carrier": "carrier-b-mobile",
"codec_chain": ["amr-wb", "g711-pcmu"],
"audio": {"sample_rate_hz": 8000, "carrier_offset_ms": 212}
Diagnosis: the bake-off ranked vendors on a channel Hazel does not use. On T01, Ryan's voice went through AMR-WB on his mobile leg and was transcoded to G.711 mu-law at the trunk; the bake-off had neither step. What a voice agent trace contains defines the carrier, codec_chain and audio fields.
What does a phone line remove?
A narrowband call is sampled at 8 kHz and passes roughly 300-3400 Hz. Nothing above 4 kHz can exist in it and the lowest band is filtered out too. A model trained only on 16 kHz speech meets input unlike its training data, and on the same words its error rate can come close to doubling.
At 8,000 samples a second the highest frequency the signal can hold is 4 kHz, and the telephone filter stops near 3.4 kHz. Much of the energy of "s", "sh" and "f" sits above that line, so these sounds blur together on a call. For a collections agent the cost lands on number words: "fifty" and "fifteen" differ only in their second syllable, so whatever the channel strips from that syllable is the difference between two amounts. Even when the handset leg is wideband, as AMR-WB was on T01, the G.711 hop brings the audio back to 8 kHz before the STT hears it.
The size of the effect is on record. In a Microsoft voice-search study, a deep neural network trained on wideband audio alone scored 27.5% WER on 16 kHz test data and 53.5% on the same data downsampled to 8 kHz; with 8 kHz data added to training it recovered to 29.3% 1.
Figure: A schematic, not a measurement. Synthetic Steelyard data. The tinted band is all a call delivers; the gray regions are removed before Hazel's STT receives a frame, and nothing downstream can restore them. In the cookbook pipeline's synthetic run, the share of energy above 3.4 kHz dropped by 65.7 dB through the band-pass and the move to 8 kHz.
Upsampling a call to 16 kHz for a model that expects it adds samples, not information. Where a vendor offers a telephony model and a general one, run both on your own phone audio; the product names do not tell you which wins on it.
Is mu-law the problem?
Mostly not. G.711 mu-law companding on its own does little audible harm; most of the damage comes from the 300-3400 Hz band and from decoding and re-encoding between codecs. Compare voices and STT models on 8 kHz G.711 audio, and treat a wideband demo as a different product.
Mu-law spends more of its 256 code values on small amplitudes, so quiet and loud callers are coded with similar relative accuracy, and ETSI TR 102 948 rates G.711 as natural sounding, with no perceptual degradation (an informal expert judgment) 2. In the cookbook pipeline's synthetic run, a phone-band signal peaking at -3 dBFS keeps 37.7 dB round-trip SNR through mu-law against 40.9 dB for plain 8-bit linear coding; at -33 dBFS mu-law still holds 34.8 dB while linear falls to 9.5 dB. Over a 30 dB drop in level, mu-law moved 2.9 dB.
Bandwidth is what hurts a demo voice: a 24 kHz TTS voice that won the listening test loses the high-frequency detail behind its lead, which can shrink on a call. The Blizzard Challenge 2009 put voices through a deliberately poor simulated phone channel, a G.712 bandpass followed by a 16 kbit/s G.726 tandem. No synthetic voice reached natural speech for intelligibility, and the organizers suggested synthetic speech may be hurt more by the channel than natural speech is 3. Tandems deserve their own test: every decode and re-encode adds error, and T01 crossed one (AMR-WB to G.711) before the STT saw a sample.
Framing is the other half. Twilio Media Streams always delivers audio/x-mulaw at 8000 Hz, mono, and requires audio sent back to be mu-law at 8000 Hz, base64 encoded 4. Tata Tele Business Services documents the same format for its Smartflo bidirectional stream and asks that audio sent to it come in 160-byte units or multiples of 160, warning that otherwise "audio gaps might occur" 5. One mu-law byte is one sample, so 160 bytes at 8,000 samples a second is 20 ms, which is RTP's default packetisation interval for G.711 6. Hazel's TTS asks for 8 kHz mu-law output directly and the media-out stage writes 160-byte frames. If your code resamples TTS audio itself, a chunk that ends partway through a 20 ms frame can put a gap on the call that no transcript-based test will ever hear.
What does packet loss do to STT?
Packet loss does real harm, but no study we found ranks it the single worst impairment. In a 2022 Frontiers study, STT error rose once loss passed 10% for a clean-trained model or 15% for a distortion-trained one, and at 0 dB SNR noise mattered more. Larger models held up longer; WER above 100% means inserted words.
Each lost packet on a G.711 call removes 20 ms of speech, and losses cluster: a cell handover or a congested hop drops several in a row. The evidence, with each study's scope:
- Where error starts to rise. Kumalija and Nakamoto (Frontiers in Signal Processing, 2022) used G.722 with no jitter buffer and no packet loss concealment. WER rose once loss passed 10% for a model trained on clean speech and 15% for one trained on distorted speech, and at 0 dB SNR noise dominated packet loss 7. For a G.711 path, read the thresholds as an order of magnitude, not a line to design to.
- What listeners lose. In a 2021 ACM TACCESS study of listeners with hearing loss, packet loss affected speech recognition, mental effort and quality ratings, while bit rate affected speech recognition for narrowband audio and quality ratings for both narrowband and wideband 8. That evidence is about the agent-to-caller direction, and about callers like P13, Ramon Diaz, 64, Spanish-dominant and hard of hearing, who already asks Hazel to repeat numbers.
- How model size changes it. Dissen et al. (Technion, Interspeech 2024) simulated loss by zeroing mel-spectrogram frames. Whisper large-v2 was "only starting to seriously degrade" beyond about 20% frame loss (French WER 7.2 at 0%, 10.3 at 20%, 27.5 at 40%), while Whisper base degraded straight away (24.7 at 0%, 48.6 at 20%, 104.9 at 40%) 9. Zeroed spectrogram frames are not RTP loss passed through a codec's own concealment, so carry over the direction of the result, not its numbers.
A WER of 104.9 can only happen through insertions: the model produced more wrong words than the reference contains. The paper does not use the word hallucination; that reading is this page's. For Hazel an inserted number is a payment amount nobody offered, so under heavy loss count hallucinated entities, defined in entity error rate, before you look at WER.
Simulate loss as bursts on 20 ms frames at several rates, and score each clip against the loss it actually received, not the rate you asked for.
Does packet loss concealment help STT?
Not reliably. Packet loss concealment (PLC) fills missing frames so a person hears smoother audio, but in the ICASSP 2024 PLC challenge the best-rated systems raised word accuracy by 0.01 at most, and one lowered it sharply. Any concealment in front of an STT is judged by word accuracy on your own calls.
PLC makes up audio at the receiver for frames that never arrived (repeat the last good frame, fade it, or generate a continuation) and is tuned for human ears, which accept a plausible guess.
The ICASSP 2024 Audio Deep Packet Loss Concealment Grand Challenge (Microsoft; 600 real loss traces, full-band audio, ITU-T P.804 crowd ratings plus word accuracy) measured both goals on the same systems. The two co-winners lifted overall quality from 2.37 for the unconcealed lossy audio to 3.49 and 3.44. Their word accuracy was 0.81 and 0.84, against 0.83 for the lossy input, and the system placed eighth cut word accuracy to 0.64 10. Concealment that pleases listeners can hurt recognition, because a convincing invented frame is still invented and the STT cannot tell which 20 ms are real.
Where your media stack allows it, test the concealment for the caller's ear and the concealment in front of Hazel's STT as two settings. And a vendor saying its stack "handles packet loss" tells you nothing about your amount entity error rate until your own loss traces have gone through it.
Why do codec effects differ by language and corpus?
The cost of a codec depends on which acoustic cues a language and a speaking style lean on, and on how each model was trained. In a 2026 study, G.711 cost English conversational audio less than one WER point on average and Japanese far more; on English read speech one model lost 42.5 points while six lost under one.
WildASR (Tay et al., Boson AI, March 2026) downsampled audio to 8 kHz, ran it through G.711 mu-law or GSM, and resampled it to 16 kHz. Averaged over seven models, error rose by +0.6 WER on English conversational audio (MagicData), +6.3 WER on English read speech (FLEURS), +9.4 CER on Japanese conversational audio, and by as much as +28.0 CER for GSM on Korean conversational audio. The English FLEURS G.711 figure comes almost entirely from one model: Qwen2-Audio lost 42.5 WER points and the other six lost 0.6 or less 11.
Three lessons:
- A codec penalty belongs to a (language, model) pair. English conversation predicts little about Spanish-dominant or Spanglish callers.
- Averages across models hide the one that fails. Report every vendor per condition, never a pooled "telephony penalty".
- No published table covers your chain. WildASR has no AMR, Opus or packet loss, and this page found no table for T01's AMR-WB to G.711 path.
The only defensible number for Steelyard's path is one measured on it, split by language group. Code-switch density and the STT language setting are their own slices, owned by normalization and code-switching in multilingual STT.
For real telephone Spanish, the nearest public set is Fisher Spanish Speech (LDC2010S01): about 163 hours of 8 kHz mu-law two-channel audio from 136 native Spanish speakers in the continental US and Puerto Rico, licensed through LDC, not free 12. Use it to check that your simulator degrades Spanish the way a phone line does; it is conversation between people, not collections calls, so it never replaces your holdout.
How do you build the phone path offline?
Start from consented 16 kHz recordings. Add noise at a fixed SNR, band-pass to 300-3400 Hz, run the access codec (AMR-NB, AMR-WB or Opus), transcode to 8 kHz G.711 mu-law, drop 20 ms frames in bursts, and stream the result to the STT in production mode. Use ITU-T reference code for any number you will defend.
flowchart LR
A["Consented 16 kHz clip<br/>(English or Spanish)"] --> B["Noise at fixed SNR<br/>(at the handset)"]
B --> C["Band-pass 300-3400 Hz"]
C --> D["Access codec encode and decode<br/>AMR-NB, AMR-WB or Opus"]
D --> E["Transcode to G.711 mu-law<br/>8 kHz, 160-byte frames"]
E --> F["Burst loss on 20 ms frames<br/>Gilbert-Elliott"]
F --> G["STT in production streaming mode<br/>English-Spanish, language setting logged"]
G --> H["Canonicalize and score<br/>entity error rate"]
Stage order is part of the model. Room noise reaches the handset before any codec, so noise mixed in after decoding is cleaner than any real borrower's car; loss happens on the network after encoding and removes whole coded frames. For an AMR-WB or Opus leg, keep the access codec at its own sample rate and let the G.711 transcode do the move to 8 kHz, because that is where a real call loses its upper band.
Bursts come from a two-state Gilbert-Elliott chain on 20 ms frames. In the good state frames arrive; in the bad state they are lost. If p is the chance of moving into the bad state and r the chance of leaving it, the long-run loss rate is p/(p+r) and the mean burst length is 1/r frames. Set r = 0.5 for 40 ms mean bursts and solve for p: a 1% loss target needs p = 0.0051, 5% needs p = 0.0263, and 10% needs p = 0.0556.
| Degradation | What it simulates | Parameters | Which metric moves | Source |
|---|
| Band-pass and 8 kHz resample | The telephone channel after the G.711 hop | 300-3400 Hz, 8,000 samples a second | STT error on fricatives and close number words ("fifty", "fifteen"); TTS clarity | Yu et al. 2013 |
| G.711 mu-law | Trunk and media stream coding | mu = 255; 160-byte, 20 ms frames | Little on its own; check frame alignment | ETSI TR 102 948 |
| Access codec plus transcode | The mobile or VoIP leg, then the trunk | AMR-NB -> G.711, AMR-WB -> G.711 (T01's chain), Opus -> G.711 | STT error of unknown size per language and model | No published table for these chains |
| Burst packet loss | Cell handover, congested hops | Gilbert-Elliott at 1%, 5%, 10%, mean burst 40 ms | Deletions first, then insertions under heavy loss | Kumalija and Nakamoto 2022; Dissen et al. 2024 |
| Concealment | The receiver's PLC | Zero fill, repeat, model-based | Word accuracy can fall while quality rises | ICASSP 2024 PLC challenge |
| Noise at fixed SNR | Car, TV, street, open office | Clean, 20 dB, 10 dB | STT error; outweighs loss at 0 dB SNR | Kumalija and Nakamoto 2022 |
The phone-path degradation recipe is a few hundred lines of numpy for learning; vendor reviews need reference code. ITU-T G.711 is still the 11/1988 text; Appendix I (09/1999) gives a standard concealment algorithm and Appendix II (02/2000) covers comfort noise 13. The ITU-T Software Tool Library (G.191), open on GitHub with v2024 (2 September 2024) as its latest release, has reference C code for G.711, the Appendix I concealment (g711iplc), the error insertion device (eid), MNRU, the sv56 level meter and the FIR and IIR filters, and is a more defensible phone-path simulator than ad hoc code 14. Its source has no AMR-NB, AMR-WB or Opus, so each access codec needs its own licensed or reference build. Until you have one, mark those rows of the matrix "not run" rather than filling them with an approximation.
What does one amount utterance look like through five conditions?
The cookbook pipeline turns one 4-second clip into five files: the 16 kHz source, the phone path, plus 5% burst loss, plus noise at 10 dB SNR, and both. Its checks confirm the upper band is gone and 200 whole 20 ms frames, and this clip's loss draw took 6 frames, 3% against a 5% target.
The global run prints this (synthetic Steelyard data, seed 20261121):
| File | Condition | What the run shows |
|---|
| 00_source_16k.wav | 16 kHz source | Energy above 3.4 kHz at -2.9 dB of the total |
| 01_phone_mulaw.wav | Band-pass, 8 kHz, mu-law | That share is -46.0 dB after mu-law; 32,000 bytes = 200 frames of 160 bytes |
| 02_phone_mulaw_loss5.wav | Plus 5% burst loss | 6 of 200 frames lost (3%), in bursts of 1, 1, 1 and 3 frames |
| 03_phone_mulaw_snr10.wav | Plus noise at 10 dB | Realized SNR 10.00 dB |
| 04_phone_mulaw_snr10_loss5.wav | Both | Same loss mask, same noise |
Figure: Synthetic Steelyard data, seed 20261121. The mask drawn is the one the recipe applied to files 02 and 04. A single 4-second clip rarely gets the loss rate you asked for, which is why the realized mask is logged with every file.
This page prints no transcripts: the course has no approved speech clip yet, and a chirp has no words. Record your own consented amount line, for example "two hundred fifty dollars by the twentieth", or for a Spanish-dominant row "doscientos cincuenta dolares para el veinte" [two hundred fifty dollars by the twentieth], and pass it through the same script. Then fill in two columns per file: what the STT wrote, and whether the canonical amount (25000 cents) survived.
Study the loss row first. The clip lost 3% against a 5% target, while the 10-seed long run landed on 5.02%: one short clip is a poor sample of a loss process. The longest burst was 60 ms. At a typical speaking rate of 4.4 syllables a second (the campaign's median agent rate in T10; the course has no measured rate for borrowers) a syllable lasts about 227 ms, so the burst covers about a quarter of one, which is small until it lands on the second syllable of "fifty", the only place it differs from "fifteen". Where a burst falls decides whether the clip loses a breath or the amount, so log the realized loss mask with every file and run many clips per condition.
How do you test the live path?
Run real SIP test calls through Linux netem with burst loss, delay and jitter on the media path. Offline frame dropping tests the recognizer; a live call also exercises the jitter buffer, the shipped concealment, voice activity detection and barge-in. A release needs both.
Linux netem supports Gilbert-Elliott burst loss with gemodel PERCENT [ R [ 1-H [ 1-K ]]] (the chance of entering the bad state, of leaving it, and the loss probabilities in the bad and good states) and draws delay jitter from uniform, normal, pareto or paretonormal distributions 15. The 5% offline setting carries over:
# on the test host interface that carries the call's RTP media
sudo tc qdisc add dev eth0 root netem loss gemodel 2.63% 50% 100% 0%
sudo tc qdisc change dev eth0 root netem delay 40ms 20ms distribution normal loss gemodel 2.63% 50% 100% 0%
sudo tc qdisc del dev eth0 root netem
What only a live call can show:
- The jitter buffer. A buffer that grows under jitter adds delay to every turn gap; measuring voice agent latency owns that number.
- The shipped concealment. Your media stack's PLC runs, not your model of it.
- Voice activity detection. A lost burst looks like a pause, so the turn detector can commit early on it. Count early jump-ins under loss; turn-taking, barge-in and recovery defines them.
- Barge-in. The barge-in monitor hears the caller through the same damaged path, so the time to stop TTS under jitter is a separate number to record.
Offline files test the STT on thousands of clips; live calls test everything around it on tens. Neither stands in for the other.
Which audio quality metrics apply to which direction?
Split by direction. Audio Hazel sends to the caller is judged by what a listener hears, so listening metrics fit. Audio the caller sends to the STT is judged by what the recognizer gets right, so task metrics such as entity error rate fit. POLQA is licensed, PESQ is withdrawn, and STOI hardly sees narrowband damage.
Never score the inbound direction with a listening metric: in the PLC challenge above, listener scores rose while word accuracy stayed flat or fell.
| Direction | Question | Metrics that fit | Owner |
|---|
| Caller -> STT | Did the recognizer get the words, amounts and dates right? | Amount and date entity error rate, hallucinated entities, WER as a tie-breaker | Entity error rate |
| Hazel -> caller | Can the caller hear and understand it, in English or Spanish? | Listening MOS predictors, listening panels, spoken-entity round trip on the phone path | Evaluating speech output for agents |
Intrusive listening metrics compare a degraded file with its clean reference, with conditions:
- POLQA (ITU-T P.863). The edition in force is still 03/2018 (reposted 12 October 2021), with Amendment 1 (04/2020) for acoustically recorded narrowband speech 16. POLQA is a patented, licensed algorithm with OPTICOM as the sole licensing agent 17, so an open lab cannot ship it. It predicts a listening MOS, not recognition accuracy, so it belongs on the Hazel-to-caller side.
- PESQ (ITU-T P.862). ITU-T lists the whole P.862 family as withdrawn or superseded; the catalog page records P.862, P.862.1, P.862.2 and P.862.3 as deleted on 5 January 2024 and points to P.863 18. It remains in wide use: Torcoli, Halimeh and Habets count over 4,600 Google Scholar results for PESQ in 2024 against a little more than 100 for POLQA, and find that PESQ versions differ by an RMSE of 0.56 to 0.62 from the ITU wideband reference when Corrigendum 2 is missing 19. Compare PESQ numbers only when version, implementation and sample rate are all stated.
- STOI. The reference Python STOI (pystoi 0.4.1) resamples both signals to 10 kHz and analyzes 15 one-third-octave bands from 150 Hz, so its top band ends near 4.3 kHz and the 300-3400 Hz phone band removes only a few bands at each end. It also rejects signals of different lengths 20. STOI is a weak detector of narrowband damage, needs time-aligned simulator output, and is no substitute for word error rate.
Non-intrusive MOS predictors and panels belong to the TTS chapter; run any of them on phone-path audio in each language Hazel speaks.
Which conditions belong in the Steelyard matrix?
Five factors: carrier, codec chain, packet loss, SNR and language group, for 4 x 3 x 3 x 3 x 3 = 324 cells. Steelyard runs 36 rows: each carrier x codec chain x loss combination once, with SNR and language group set so every pair of levels across any two factors appears at least once. A script confirms all 102.
| Factor | Levels |
|---|
| Carrier | 3 US mobile carrier profiles (T01's carrier-b-mobile is one) and 1 VoIP leg |
| Codec chain | AMR-NB -> G.711; AMR-WB -> G.711; Opus -> G.711 |
| Packet loss | 1%, 5%, 10% (Gilbert-Elliott, 40 ms mean bursts) |
| SNR | Clean, 20 dB, 10 dB |
| Language group | English, Spanish-dominant, Spanglish |
A carrier profile is a level only if it behaves differently from the others; build each from live test calls on that network (measured loss, jitter, level and negotiated codec modes). The carrier factor carries network behavior and the codec chain is separate, so a row such as the VoIP leg with AMR-NB is a stress condition, not a common call; keep it, since a carrier can change what it negotiates without notice.
The design indexes levels from 0, runs each carrier x codec chain x loss combination once, and sets SNR to level (codec + loss) mod 3 and language group to (codec + 2 x loss + carrier) mod 3. Of the 10 factor pairs, the 4 with the carrier have 12 level pairs each (48) and the other 6 have 9 each (54): 102 in all, and a brute-force check over the 36 rows finds every one. Each carrier appears in 9 rows and each level of every other factor in 12.
What the covering design gives and what it does not:
- Pairwise interactions are covered. If Opus to G.711 fails only on Spanish-dominant callers, some row has that pair; a one-factor-at-a-time plan would not.
- Cells are not estimates. Gate on pooled factor levels (all 12 rows at 10% loss, all 9 on the VoIP leg) and use cells to locate a failure, never to certify one.
- Clips are paired. Each row draws clips in its language group, and within a group the same clips go through every row.
- Three-way interactions are not covered. AMR-NB at 10% loss and 10 dB SNR may never share a row; the real holdout exists partly for that.
For two vendors on the same clips, use the paired difference in amount entity error rate, bootstrapped by speaker; bootstrap and paired tests has the method.
What gate would have caught it?
A rule that STT and TTS gates run only on phone-path audio. The bake-off would then have scored the vendors on the 36-row matrix and a real 8 kHz holdout from consented calls, with 16 kHz scores in a column that never gates, and vendor A would have finished third before the shortlist, not after T01.
gate: stt-phone-path
audio:
simulated: consented 16 kHz volunteer clips (English and Spanish) through phone-path v1, 36-row matrix
real_holdout: gold-v7 STT set (1,200 reference-transcribed caller utterances, 180 speakers, 8 kHz, stratified by carrier)
wideband: reported in its own column; never gates
metric: amount and date entity error rate per language group (English, Spanish-dominant, Spanglish), hallucinated entities alongside
interval: speaker-clustered bootstrap, 95%
block_if:
- real_holdout upper_bound > campaign_threshold
- any factor level (pooled rows) upper_bound > campaign_threshold
- paired_difference_upper_bound > margin # candidate minus incumbent, same clips, by speaker
report_always: [per-cell results, realized loss per clip, STT language setting, rows not run (access codecs without a reference build)]
On intervals: utterances from the same speaker are correlated, so use a bootstrap clustered by speaker rather than Wilson, and a paired bootstrap by speaker for the difference between two vendors. The thresholds and the margin are the campaign owner's numbers, as set out in entity error rate.
A simulator holds only the impairments someone thought to model. The real holdout catches the rest (a handset's noise suppression, an undocumented transcode) and is the ground truth for the simulator: when the two rank vendors differently, believe the holdout and fix the simulator.
TTS follows the same rule: the spoken-entity round trip in evaluating speech output for agents runs on 8 kHz mu-law, in en-US and es-US.
What goes in the decision memo?
The memo pins down phone path v1: the degradations and their levels, what runs offline and what runs live, which holdout, and the rule that wideband scores never gate. Steelyard's version follows.
Decision. Every STT and TTS gate uses phone path v1: the 36-row matrix offline, netem-impaired SIP test calls before each release, and the gold-v7 STT set as the real 8 kHz holdout. Wideband scores are reported and never gate.
Context. The 16 kHz bake-off put vendor A first; scored by amount entity error rate on mobile-to-trunk audio (AMR-WB to G.711) it came third of four. Production audio is 8 kHz G.711 mu-law in 20 ms frames, after an AMR-NB, AMR-WB or Opus access leg (T01: AMR-WB).
Options considered. (1) Keep the wideband bake-off: cheap, and it picked the wrong vendor. (2) Real calls only: true audio, no control over conditions, slow to grow, thin in Spanish. (3) Simulated matrix plus real holdout plus live netem calls: chosen.
Evidence. Wideband-only training nearly doubled WER at 8 kHz (Yu et al. 2013); codec penalties varied by language and model (WildASR, 2026); concealment raised quality without raising word accuracy by more than 0.01 (ICASSP 2024). Gates use speaker-clustered 95% intervals on gold-v7, per language group.
Owner and cadence. The eval platform team owns the path; the STT owner runs it on every vendor model version and every quarter; carrier profiles are rebuilt quarterly from live test calls.
Tripwires. The matrix and the holdout rank vendors differently; a carrier or codec chain appears in traces that the matrix lacks; access-codec rows are still not run at the next release.
Dated facts. WildASR v1 (March 2026); ITU-T P.863 page (April 2026); STL v2024; Fisher Spanish catalog entry read 20 November 2026.
What to do this week
- Count every
carrier and codec_chain value in one week of call traces, including VoIP legs.
- Rerun your most recent STT comparison on 8 kHz mu-law audio cut into whole 20 ms frames, and put the 16 kHz column beside it.
- Add burst loss at 1%, 5% and 10% with 40 ms mean bursts, record the realized loss per clip, and count hallucinated entities in the worst condition.
- Place one SIP test call through netem and listen for gaps, early cutoffs and missed barge-ins, once in English and once in Spanish.
- Write "wideband scores never gate" into the release checklist.
For the general method behind perturbation suites, see perturbation and noise testing; metric targets for voice are in the voice playbook.
NOTE
What will date on this page. WildASR is arXiv 2603.25727 v1 (26 March 2026); recheck for a version that adds AMR, Opus or packet loss. The ITU-T P.863 page showed the 03/2018 edition in force when last updated on 24 April 2026; the P.862 family was deleted on 5 January 2024. The ITU-T STL's latest release is v2024 (2 September 2024). The tc-netem page was read on 20 November 2026; pystoi is 0.4.1 (29 December 2023). Twilio's and Smartflo's stream formats and the LDC catalog entry for Fisher Spanish were read on 20 November 2026 and can change without notice. Recipe numbers are tied to the global seed 20261121.