What is speaker diarization?

September 24, 2026

If you've seen a transcript with S0 and S1 in front of each line and wondered where those labels come from, the answer is diarization. It's worth understanding before you build anything on top of it, because it promises less than people assume.

What it is, in one sentence

Speaker diarization is the process of splitting a recording into stretches and deciding how many distinct voices there are and when each one speaks. The question it answers is exactly that: who speaks, when. Not a word more.

What you get back looks like this:

S0 — Good morning, I'm calling about order 4471. S1 — One moment, let me check. S0 — No rush.

S0 and S1 aren't names. They're speaker labels, local to that recording. That's the confusion worth clearing up straight away.

How it works

Diarization doesn't understand what's being said; it works on the sound. Roughly, the process has three steps:

• Segmentation: the audio is cut into stretches where a single voice is active, by detecting silences and turn changes.
• Voice embedding: from each stretch it extracts a vector summarising that voice's acoustic characteristics — pitch, timbre, cadence. It's a numeric representation, not an identity.
• Clustering: stretches with similar embeddings are grouped together. Each group becomes a speaker and gets its label.

So it's an acoustic clustering problem, not recognition. The system doesn't compare your voice against any database: it only looks at which stretches of this recording resemble each other. Diarization is often combined with transcription, so each transcribed line ends up assigned to its label.

When you need it

Almost any time you have more than one person on the same track and want something more than a block of text:

• Phone system calls: separating customer and agent turns a wall of text into a readable dialogue and lets you measure who spoke for how long.
• Meetings and interviews: knowing the turns lets you find one specific contribution inside hours of recording.
• Minutes and summaries: a summary comes out much better when it knows who said what, rather than blending everything into one voice.

If your audio is a single person — a dictation, a voicemail, a voice note — you don't need diarization at all. There it only gets in the way.

Where the limits are

Here's what diarization does not do, and it's worth being clear before you promise it to anyone:

• It doesn't know who S0 is. There are no names behind it, and it won't invent any. Putting a name to a label is on you, using context you already have: the call metadata, the order of speaking, or the audio itself when someone introduces themselves.
• Labels don't carry across recordings. Tuesday's S0 needn't be Wednesday's S0. They're local to each file.
• It isn't infallible. Two similar voices can merge into a single label, and a voice that shifts register a lot can split into two. The number of speakers it detects is an estimate.

And there's a line worth not crossing: identifying a person by their voice is biometric data processing, which the GDPR puts in special categories and which demands a completely different regime. Diarization isn't that, and almost nobody who asks for it actually needs to go that far.

In short

Diarization gives you the turn-taking of a conversation: how many voices and when each one speaks. It's a very useful tool when you use it for what it is, and an expensive disappointment when you expect it to tell you who people are.

In Uttera you have it alongside the rest of the audio flow — transcribe, translate, summarise, and know who speaks, over the same recording — and you can see how it fits in the use cases. The machine puts the labels; the names come from the context you already have.

Anything to add or correct? Write to support@uttera.ai. If you correct us, we edit the post and credit you.

← All posts