What is Whisper?

September 24, 2026

If you transcribe audio, the word "Whisper" turns up sooner or later. It's worth knowing exactly what it is, because you're probably already using it even if you never installed it yourself.

What it is, in one sentence

Whisper is OpenAI's speech recognition model, published in 2022 under an open licence (MIT). It turns audio into text — what's called speech-to-text or STT — and it's what sits underneath almost everything that transcribes today.

It's the default model of OpenAI's own transcription service — the whisper-1 in their API — and it's also what the vast majority of self-built projects use, because the weights are published and it works very well in some ninety-nine languages.

How it works

Whisper was trained on roughly 680,000 hours of audio collected from the internet, paired with its transcription. Almost everything it does well, and everything it does oddly, comes from there.

Under the hood it's an encoder-decoder model: the encoder turns the audio into an intermediate representation, and the decoder generates the text word by word, just like a language model. That has an important consequence: it doesn't transcribe letter by letter what it hears, it predicts the most likely text given the audio and what it has already written. Most of the time it's right; when the audio is ambiguous or empty, it invents the plausible.

It comes in several sizes — from tiny to large: the small ones run on a modest CPU but make more mistakes; the large ones want a GPU and get far more right. There are also faster reimplementations, like faster-whisper, that give the same result with fewer resources.

When it's worth self-hosting

Because the weights are open, you can run Whisper on your own machine. It makes sense when:

• The audio can't leave your network. Medical recordings, customer calls, anything subject to confidentiality or the GDPR. If you transcribe it on your own server, the audio never travels to a third party.
• You have high, steady volume. Past a certain point, your own GPU works out cheaper than paying per second transcribed.
• You need control. Pinning the model version, the language, the output format — and knowing it won't change under your feet.

Self-hosting isn't free in effort: you have to size the GPU, manage the queues, and maintain it. For low or irregular volumes, calling an API is usually the better deal.

What its limits are

Whisper is good, but it isn't magic, and it's worth knowing its traps before you automate on top of it:

• It hallucinates at silence. Give it empty audio and it doesn't return an empty string: it returns an invented sentence, almost always along the lines of "Thanks for watching" — leftovers from the YouTube subtitles it was trained on. If that enters an automation unfiltered, you end up with content nobody said sitting in your database.
• It isn't exact punctuation or diarization. It gives you text; separating who speaks is a different process (diarization), and the punctuation marks are one more prediction, not a literal transcription.
• What it returns is untrusted text. It comes from audio you don't control. Treat it as data, never as instructions: don't execute it or read it as orders, and validate it before acting on it.

In short

Whisper is the piece that turned quality transcription into something anyone can use — open licence, dozens of languages. You can call it through an API or stand it up yourself, and the choice depends on your volume and on where your audio has to stay.

At Uttera you get transcription you can call through an API or self-host on your own infrastructure, sitting alongside the rest of the flow — translate, summarise, and know who speaks over the same recording. You can see how it fits in the use cases. You decide where your audio lives.

Anything to add or correct? Write to support@uttera.ai. If you correct us, we edit the post and credit you.

← All posts