A local Whisper transcription pipeline: picking the right model size

There’s a reflex, when you need to turn audio into text, to reach for a hosted API. It’s understandable: speech recognition used to be hard, and paying someone else to do it felt like the safe choice. But that reflex is out of date. OpenAI’s Whisper is open, the weights are free, and a small reimplementation called faster-whisper makes it quick enough to run on a plain CPU. You can transcribe an hour of audio on your laptop, offline, for nothing.

The catch is choosing. Whisper ships in several sizes, from a 39-million-parameter tiny model up to a 1.5-billion-parameter large, and the gap between them in speed and accuracy is enormous. The pipeline is short: ffmpeg to prepare the audio, faster-whisper to transcribe it. Picking the tier is the part that takes judgment.

Free, offline, and good enough

Whisper came out of a 2022 paper, Robust Speech Recognition via Large-Scale Weak Supervision, whose central result is that “when scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks.” That scale is why Whisper handles accents, background noise, and code-switching far better than the open models that came before it, and why a base-sized Whisper is genuinely useful rather than a toy.

Running it locally used to mean the reference PyTorch implementation, which is accurate but heavy. faster-whisper reimplements the same models on top of CTranslate2, an inference engine for transformer models, and the result “is up to 4 times faster than openai/whisper for the same accuracy while using less memory.” For a free, offline tool that runs on whatever hardware you happen to have, that speedup is the difference between practical and not.

Preprocessing audio with ffmpeg

Whisper has firm expectations about its input: mono audio at 16 kHz. The reference code hard-codes it (SAMPLE_RATE = 16000) and feeds everything through a mel-spectrogram front end built around that rate. faster-whisper will resample for you internally, but doing it up front keeps the input small, predictable, and decoupled from whatever container your source happened to arrive in.

ffmpeg does the whole job in one line. Here’s the extraction step from the pipeline: take any video or audio file, down-mix to a single channel, resample to 16 kHz, and drop the video stream:

ffmpeg -y -i "$VIDEO" -ac 1 -ar "$SRATE" -vn "$OUTPUT"

The three flags are the entire transformation: -ac 1 collapses stereo (or 5.1) to mono, -ar 16000 resamples to the rate Whisper wants, and -vn strips the video so you’re not carrying around frames you’ll never use. The full script just wraps that with directory handling and a default sample rate:

VIDEO="$1"
SRATE="${2:-16000}"
AUDIO_DIR="$(dirname "$(realpath "$0")")/../audio"
mkdir -p "$AUDIO_DIR"

BASENAME="$(basename "$VIDEO")"
NAME="${BASENAME%.*}"
OUTPUT="$AUDIO_DIR/${NAME}.wav"

ffmpeg -y -i "$VIDEO" -ac 1 -ar "$SRATE" -vn "$OUTPUT"

Collapsing to mono costs nothing here: Whisper is a mono model, so there is nothing to lose, and a 16 kHz mono WAV is a fraction of the size of the original.

Quantization and model tiers

Once the audio is ready, transcription is a few lines. The one decision baked into this snippet is compute_type="int8":

from faster_whisper import WhisperModel

# base: good quality/speed tradeoff; use "small"/"medium" for better quality
model = WhisperModel("base", device="cpu", compute_type="int8")
segments, info = model.transcribe(str(audio_path), beam_size=1)

CTranslate2 supports 8-bit integer (INT8) quantization, which stores the model’s weights as 8-bit integers instead of 32-bit floats. The model gets roughly four times smaller in memory and runs faster, because integer arithmetic is cheaper than floating-point and far more of the model fits in cache. faster-whisper’s own benchmarks show the small model on CPU dropping from 2257 MB at full precision to 1477 MB with int8, a real saving on a laptop. The accuracy cost is small enough that int8 is the sensible default for CPU transcription.

beam_size=1 is the other speed lever: greedy decoding instead of a beam search. It’s faster and, for clean audio, usually indistinguishable in quality. The segments come back as a stream you iterate over, each one a chunk of text with a start and end timestamp:

segment_list = []
full_parts = []
for s in segments:
    segment_list.append({"start": s.start, "end": s.end, "text": s.text.strip()})
    full_parts.append(s.text)

result = {
    "text": " ".join(full_parts).strip(),
    "language": getattr(info, "language", "en"),
    "segments": segment_list,
}

You get both a flat transcript (text) and the timestamped segments, which is what you’d use to build subtitles or to jump to a moment in the source.

Measuring latency vs. accuracy

So which model do you load, tiny or large-v3? I stopped guessing and measured it. Everything below ran on one machine, a 13th-gen Intel Core i7 with 16 threads, on CPU, using the exact pipeline config: compute_type="int8", beam_size=1. Two numbers per model: how long it takes, and how wrong it gets.

Latency first. I stitched a fixed 70-second clip together and timed each model transcribing it, median of three runs. Transcription time tracks the parameter count, and the model card makes the spread concrete: tiny is 39 M parameters, base 74 M, small 244 M, medium 769 M, large-v3 1550 M. On a CPU that turns into wall-clock time you feel:

tiny clears the whole clip in 1.7 seconds, 41 times faster than real time. large-v3 needs 49.6 seconds, barely ahead of the audio itself. The steep part is the top end: small, medium, large run 13, then 33, then 50 seconds, each step close to double the one before.

Accuracy uses word error rate (WER), WER = (S + D + I) / N: substitutions, deletions, and insertions over the reference word count. Lower is better; 0.05 means about one word in twenty is wrong. I scored all five models on 200 utterances of LibriSpeech test-clean, a standard read-speech set that ships ground-truth transcripts, using jiwer with the usual case- and punctuation-insensitive normalization.

Then the curve did something I wasn’t expecting. It drops the way you’d hope from tiny to medium, 7.7% down to 2.9%, and then it climbs back up:

large-v3 scored 4.7%. Worse than medium, worse than small. The 1.5-billion-parameter model lost to one a fifth its size, on clean read speech, while taking twice as long to do it.

I didn’t trust that, so I broke it down per clip. The median large-v3 transcript is perfect: at least half the utterances score a WER of exactly zero. The aggregate is wrecked by a few disasters. Two clips came in above 50%, and the worst one is diagnostic. large-v3 transcribed a 16-word line correctly (“the sugar manufacturer who says loaf clarified lumps…”) and then refused to stop, inventing a sentence about a membrane before falling into a loop of the single word “Why,” over and over. That one clip scored 325% on its own. Drop it and the aggregate falls to 3.4%.

That runaway is the Whisper hallucination loop, and greedy decoding opens the door to it. beam_size=1 commits to the single likeliest next token at every step with no way to abandon a bad path, and the largest, most fluent model is the readiest to confabulate a fluent continuation past the end of the audio. I reran that clip to check: greedy spun out to 161 words; beam_size=5 brought it back to 25. The accuracy was never the model’s problem. It was the fast setting I had bolted onto it.

Which Whisper model size should you use?

Put the two curves side by side and the call almost makes itself, though not the way I assumed walking in.

• tiny / base: live or near-real-time work, quick drafts, batch jobs over thousands of files where throughput is everything. base is this pipeline’s default for a reason: fast, light, fine for searchable transcripts and rough subtitles.
• small: the value pick. 4.0% WER for 13 seconds on the clip, a real jump over base you can absorb on a single file.
• medium: the most accurate model in the run at 2.9%, when you can pay roughly real-time speed for the transcript.
• large-v3: not the automatic “best.” At the pipeline’s greedy default it was both the slowest model and the least accurate of the top three. If you do reach for it, switch on beam search first, or you are paying for 1.5 billion parameters and buying hallucinations.

The old rule of thumb comes out stronger: start one tier smaller than you think you need, read the output, and move up only if the errors actually hurt. People reach for large on reflex. Here that meant waiting nearly four times as long as small for a transcript that measured worse.

Where I used this

I built this pipeline as yt-transcribe, a small download-extract-transcribe tool for pulling text out of copyright-cleared talks and lectures. It runs entirely offline, no API key, with base as the default and a one-line edit to move up a tier when a particular recording needed it. The shell-and-Python shape kept it easy to drive by hand or from a script.

The lesson that travels beyond this one tool: capable speech recognition is now free and local, and the engineering isn’t in calling an API. It’s in the preprocessing (mono, 16 kHz), in matching the model tier to your latency and accuracy budget, and, as the benchmark reminded me, in the decode settings underneath. Get those right and you rarely need to pay anyone for transcription again.

References and further reading

• A. Radford et al., “Robust Speech Recognition via Large-Scale Weak Supervision”: the Whisper paper; arXiv:2212.04356, the 680,000-hour weak-supervision result. (Link verified.)
• faster-whisper: Whisper reimplemented on CTranslate2; “up to 4 times faster than openai/whisper … while using less memory,” with int8 CPU benchmarks. (Link verified.)
• CTranslate2: the inference engine behind faster-whisper; supports INT16/INT8/INT4 quantization on CPU and GPU. (Link verified.)
• OpenAI Whisper: available models and languages: the model card table with parameter counts (tiny 39 M → large 1550 M). (Link verified.)
• Word error rate: the (S + D + I) / N metric used to compare transcription accuracy. (Link verified.)
• LibriSpeech ASR corpus: the read-speech benchmark; I scored WER on the test-clean split (200 utterances). (Link verified.)
• jiwer: the library used to compute WER with case- and punctuation-insensitive normalization. (Link verified.)
• FFmpeg resampler documentation: the official docs for sample-rate and channel-layout conversion (-ar, -ac). (Link verified.)