Robust Speech Recognition via Large-Scale Weak Supervision

By 2022, speech recognition models had beaten the reported human error rate on LibriSpeech, a benchmark of audiobooks read aloud, by a wide margin. The same models stumbled on phone calls, meetings, accents and noisy rooms. The authors, Alec Radford, Jong Wook Kim and colleagues at OpenAI, argue that this gap comes from how models are trained: tuned on one dataset, they learn its quirks, which helps on that dataset's test set and nowhere else.

Whisper trains on breadth instead. The team gathered 680,000 hours of audio that already had transcripts on the internet: 438,218 hours of English, 117,113 hours in 96 other languages, and 125,739 hours of foreign speech paired with English text. The transcripts are imperfect, which is what "weak supervision" means here, so the authors filtered out machine-generated captions, mismatched languages and low-quality sources. The model is a standard encoder-decoder transformer. It reads 30 seconds of audio and writes tokens, and a few special tokens at the start tell it which language, which task (transcribe or translate into English) and whether to emit timestamps.

The evaluation is the other half of the idea. Whisper was never fine-tuned on the benchmarks it was tested on. The largest model scored 2.7 word error rate (WER) on LibriSpeech test-clean, roughly level with a supervised wav2vec 2.0 model trained on LibriSpeech, and made 55.2% fewer errors on average across 13 other English test sets. On 25 recordings it came within about a point of professional human transcribers.

Call centers, sales calls, field service, medical dictation, meetings: a large share of enterprise AI work starts with audio. Whisper and models built on its recipe are often the first transcription layer, and this paper explains both why they work well on messy audio and how they fail.

Benchmark scores do not transfer between audio types. Two models with the same LibriSpeech score differed by more than 2× in errors on other datasets. Ask a vendor for numbers on audio like the customer's, or measure on a sample yourself.

WER depends on formatting rules. "$68 million" and "sixty-eight million dollars" count as several errors under raw WER. The authors saw WER fall by up to 50% on some datasets from text normalization alone. Agree on a normalizer before comparing systems.

Long recordings need decoding care. The model sees 30 seconds at a time. Over an hour-long call, a bad window can cascade into repeated text, skipped words or invented sentences. The paper's decoding heuristics are part of the system, not optional extras.

Hallucination is a real failure mode. The authors report complete hallucinations unrelated to the audio, and an early tendency to invent speaker names. In regulated settings, a confident wrong transcript is worse than a gap.

Language coverage follows the data. Accuracy in a language tracked how many hours of training audio it had. For a customer whose callers speak a lower-resource language, test before promising.