A call recording without speaker labels is a wall of text. Anyone reviewing a complaint or auditing a sales conversation needs to know which sentences belonged to the customer, because a phrase such as "I will waive the charge" means something entirely different depending on who said it. A summary built on a transcript that confuses the two is misleading rather than merely incomplete.
Speaker diarization supplies those labels. It has been studied for more than two decades, through NIST evaluation campaigns and research on broadcast news, meetings and telephone conversations, and it remains genuinely unsolved in the conditions that matter most. This post covers the classical pipeline, end to end approaches, how the task is measured, why telephony is the hard case, and the privacy questions raised by systems that build a mathematical representation of a person's voice.
What diarization is, and what it is not
Diarization partitions a recording in time and assigns each speech segment to a speaker cluster. The output is a set of labelled intervals: speaker A from 0.0 to 4.2 seconds, speaker B from 4.4 to 5.1, and so on.
It is not source separation: it produces time labels over the original mixture and never unmixes anything. Two further boundaries are worth drawing.
- It is not identification. The labels are arbitrary. Nothing connects speaker A in one recording to speaker A in another unless a separate identification step is added.
- It is not transcription. Diarization and speech recognition are separate tasks, usually run in parallel and then aligned. The two systems make independent errors, and a recogniser's word boundaries rarely coincide with a diarizer's segment boundaries. A labelled transcript is therefore the product of two components, and an obviously wrong label may originate from either one.
Why it matters for call records
Diarization sits underneath several things usually treated as separate features.
- Readable records. A reviewer opening a call from three weeks ago needs a dialogue, not a monologue.
- Compliance review. Verifying that a required disclosure was delivered by the representative, not paraphrased by the customer, requires attribution.
- Analytics. Talk to listen ratio, interruption counts and question counts are speaker dependent measures, and none can be computed from an unlabelled transcript.
- Extraction accuracy. When a system pulls an address or a commitment out of a conversation, whether the customer stated it or the agent proposed it changes its status entirely, and models trained on those transcripts inherit the errors as label noise.
The classical pipeline
The traditional approach remains a strong and widely deployed baseline. It is modular, so each stage is replaceable and each contributes its own failure modes.
Segmentation
The first step separates speech from non speech, then cuts speech into units short enough to belong to a single speaker. Historically this used change point detection with criteria such as the Bayesian Information Criterion. Many modern systems instead use uniform short windows and let later stages resolve boundaries, avoiding a detector that performs poorly at rapid turn exchanges.
Speaker embeddings
Each segment is converted into a fixed length vector intended to capture speaker characteristics while discarding phonetic content. Early representations included i-vectors from factor analysis and d-vectors from averaged network activations. x-vectors, introduced by Snyder and colleagues at Johns Hopkins, use a time delay neural network with a statistics pooling layer that aggregates over a variable length segment, and became the dominant representation. Later architectures such as ECAPA-TDNN improved on the same formula: train a network to discriminate among many speakers, discard the classification layer, and use the internal representation as a general purpose voice embedding.
Embeddings are compared with cosine similarity or with probabilistic linear discriminant analysis. The critical constraint is duration: embeddings become unreliable as segments shorten, and conversational speech is full of one word turns.
Clustering and resegmentation
Embeddings are then grouped. Agglomerative hierarchical clustering starts with every segment as its own cluster and repeatedly merges the closest pair until a stopping criterion is met. It is simple, interpretable, and highly sensitive to that threshold, which effectively decides how many speakers the system believes are present. Spectral clustering builds an affinity matrix, computes an eigendecomposition and clusters in the resulting space, often handling non convex cluster shapes better; its eigenvalue spectrum also helps estimate the speaker count. A final pass then reassigns frames using the clustered models, often with a Variational Bayes hidden Markov model, as in the VBx method developed at Brno University of Technology.
End to end models, overlap and unknown speaker counts
The modular pipeline has a structural weakness: its stages are optimised separately, and clustering assumes each segment belongs to exactly one speaker. Overlapping speech violates that assumption by construction, and people talk over each other constantly.
End to end neural diarization, usually abbreviated EEND and introduced by Fujita and colleagues at Hitachi, reformulates the task as multi label frame classification. The network outputs an independent activity probability per speaker for each frame, so two speakers can be active at once without contradiction. Training raises the label permutation problem: with two output channels, there is no inherent reason channel one should correspond to the first reference speaker. Permutation invariant training computes the loss under every possible assignment of outputs to references and takes the minimum, so the network is not penalised for arbitrary ordering. The technique came from earlier speech separation research. Early EEND systems handled only a fixed, small number of speakers; later attractor based formulations generate a variable number of speaker attractors from the input. The field has not converged on one architecture, and carefully tuned modular systems remain competitive.
The unknown speaker count is a related difficulty. In a two party call the answer is usually two, but a handover to a colleague, a family member taking the phone or a conference bridge breaks that assumption. Systems that fix the count fail when reality differs; systems that estimate it depend on exactly the signal that degrades when voices are similar or audio is poor.
Measuring diarization: DER and its components
Diarization error rate is the standard metric: incorrectly attributed speech as a proportion of total reference speech, after finding the optimal mapping between hypothesised and reference labels. It decomposes into three parts:
- Missed speech. Reference speech marked as non speech or left unattributed. Overlap contributes heavily for systems that cannot output two speakers at once.
- False alarm speech. Non speech labelled as speech, typically noise, echo residue or breathing.
- Speaker confusion. Speech correctly detected but assigned to the wrong speaker. This is the error most visible to a human reader.
Two cautions. DER is duration weighted, so it under weights short turns, and short turns are precisely where conversational meaning often sits: a system that misattributes every one word acknowledgement can still report a low DER. And comparisons are meaningless without matching collar settings and overlap handling, because for years the standard convention excluded overlapped regions entirely.
Why telephony is the hard case
Diarization research has often been benchmarked on meetings and wideband recordings. Telephone calls stack several difficulties at once.
- Narrow bandwidth and codec artefacts. Traditional telephony carries roughly 300 Hz to 3.4 kHz, so much of the spectral detail that distinguishes voices, particularly higher formant structure and fricative energy, is absent. Low bitrate codecs preserve intelligibility, not speaker detail.
- Short turns. Calls are dense with utterances under a second, and an embedding from a mumbled acknowledgement carries little speaker information.
- Similar voices. Two adult speakers of the same gender, similar age and similar regional accent may sit close together in embedding space, and a mid call change of handset can split one speaker into two clusters.
The channel shortcut
One case makes diarization trivial: each party recorded on its own channel. Where the recording point sits before the two directions are mixed, attribution is exact, with no model and no error.
It is not always available. Recording may happen after the legs are mixed, a transfer may change the topology mid call, or audio may arrive as a single file. Separation by channel also solves attribution between legs but not within a leg, since two people sharing a handset share a channel.
Multilingual and code mixed calls
Indian telephone conversations move between languages within a single turn. A caller might say "Haan ji, booking confirm kar dijiye, but please send the invoice on WhatsApp," and a Gujarati speaker may mix Gujarati, Hindi and English as freely.
Speaker embeddings are trained to be text independent, so in principle a language switch should not move a speaker's embedding. In practice, extractors trained predominantly on one language family may be less well calibrated on others, evaluation sets for Indian language telephony remain limited, and speakers shift pitch range, rate and formality when switching language, adding within speaker variability. Evaluation must therefore use audio that resembles actual traffic.
Privacy and voice biometrics
A speaker embedding is a compact numerical representation of a voice, computed so that it is similar for the same speaker and different for others. That is functionally a biometric characteristic. Under several data protection regimes, biometric data used to uniquely identify a person attracts stricter requirements, and India's Digital Personal Data Protection Act, 2023 governs personal data handling generally.
This is not legal advice. The analysis depends on jurisdiction and on facts specific to each deployment, so organisations should take advice rather than rely on a general article. At an engineering level, three questions are worth asking early.
- Are embeddings persisted at all? A diarizer can compute and discard them within a single recording. Storing them creates a durable asset that did not previously exist, and matching a voice across recordings is identification, not diarization.
- What is the retention period, and who has access? Embeddings should not outlive the recordings that justified them, and access controls should reflect the nature of the data rather than treating a vector as innocuous because it is not audio.
A useful default is minimisation. If the need is a readable transcript with two labels, it is met without retaining any voice representation once the labels exist.
Practical guidance
For teams building on call recordings, a short list that reliably pays off.
- Prefer per channel recording wherever the architecture allows it, keeping acoustic diarization as a fallback for mixed sources and shared handsets. No model competes with a correct answer obtained for free.
- Evaluate on your own traffic, including narrowband calls, code mixed speech and noisy environments. Published benchmark numbers do not transfer.
- Score overlap explicitly and report the DER components separately, since missed speech and speaker confusion call for different fixes.
- Measure what the reviewer sees, meaning word level attribution after alignment, not only duration weighted DER.
- Decide the embedding retention policy before building.
Diarization is progressing steadily, with overlap aware neural models and better multilingual embedding training as the most promising directions, but it remains a task where conditions dominate. At Cirio we regard accurate, well attributed call records as part of the basic product rather than an optional analytics layer, because a record nobody can trust is not a record.
Frequently Asked Questions
What is speaker diarization?
Speaker diarization is the task of partitioning an audio recording into segments according to speaker identity, commonly summarised as determining who spoke when. It does not identify people by name. It only groups speech into distinct speaker clusters and labels them arbitrarily, for example as speaker one and speaker two.
How is diarization different from speaker recognition?
Speaker recognition compares a voice against enrolled identities to verify or identify a known person. Diarization has no enrolled identities and no prior knowledge of who is present, so it clusters segments by voice similarity within a single recording. The same person may therefore receive different labels in two different recordings.
What is diarization error rate?
Diarization error rate, or DER, is the standard evaluation metric. It is the proportion of speaking time that is wrongly attributed, and it decomposes into missed speech, false alarm speech and speaker confusion. Scoring conventions vary, particularly over overlapping speech and boundary collars, so DER figures are comparable only under matching conditions.
Why is diarization harder on phone calls than on meeting recordings?
Telephone audio is usually narrowband, so the higher frequencies that help distinguish voices are missing, and lossy codecs add artefacts on top. Calls also contain many very short turns such as acknowledgements, which give too little audio to characterise a voice, and both parties may be similar in age, gender and accent.