NVIDIA Nemotron 3 Diarization is an open-weight, 100M-parameter model that classifies who spoke when in live or recorded conversations, supporting up to eight speakers and overlapping speech. It ranks #1 in VoiceArena’s initial Diarization-Bench leaderboard with a 14.72% diarization error rate (DER) across 139 English conversations totaling about 22 hours, compared with 19.3% for the next-ranked system, a roughly 24% relative reduction. The model follows the Sortformer approach: output speaker channels are ordered by arrival time, so the first new voice becomes speaker_0, the next speaker_1, and so on. This makes generic speaker labels stable across streaming chunks without solving a new permutation each time. The labels are anonymous channel IDs, not real identities; downstream applications can map them to people using meeting metadata, user profiles, or active speaker verification.

The model accepts 16 kHz single-channel audio, converts it to Mel-spectrogram features with a 10 ms frame step, stacks features by eight to produce 80 ms frames, and feeds a 31-layer Transformer encoder with rotary positional embeddings. A Conv1D layer upsamples predictions back to input resolution, producing a [T, 8] tensor of speaker-activity probabilities at a default 10 ms stride. Overlap is handled naturally because multiple channels can be active in the same frame. For streaming, two memory mechanisms provide context: the Arrival-Order Speaker Cache (AOSC) retains speaker information from earlier chunks, and a FIFO queue supplies recent frame context. The input buffer also includes right context after the current chunk; more right context improves interpretation of speaker transitions, while less reduces wait time. Chunked inference removes a fixed maximum audio duration, though performance can degrade on unusually long recordings or severe noise, reverberation, far-field capture, or domain shift.

Training used public and licensed speech data, including multispeaker-annotated real-world conversations licensed from David AI and large-scale simulated English and multilingual mixtures spanning 21 languages. Adding David AI data decreased compound DER by 0.77 absolute points, from 11.19% to 10.42%, at both offline-style and ultra-low-latency operating points. The model supports recommended input-buffer latencies of 30.4, 1.04, 0.64, and 0.32 seconds; shorter buffers respond sooner, while more context generally improves accuracy and throughput. The 0.32-second configuration is the lowest recommended, even though an 80 ms buffer is technically possible.

In NVIDIA’s own evaluation across eight public benchmarks, the final model reduces DER on all eight conditions at 1.04-second latency compared with the previous four-speaker Streaming Sortformer baseline. Relative reductions range from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM, with an unweighted mean of 41.0%. The improvement is consistent across shared latency settings, and gains widen in higher-speaker-count subsets of DIHARD III, CALLHOME-Part2, and NOTSOFAR1. One nuance: on the two-speaker CALLHOME subset, the final model records 5.98% DER versus 5.68% for the baseline, though full-set CALLHOME-Part2 DER improves from 10.32% to 9.10%. DIHARD III groups recordings with five through nine speakers into one result, so that aggregate includes audio beyond the model’s eight-speaker maximum.

Throughput results are reported as real-time factor speedup (RTFx), measured on an NVIDIA RTX PRO 5000 with BF16 and the NeMo PyTorch backend. At the 30.4-second configuration, Nemotron 3 Diarization reaches 15,113x RTFx at batch size 32 with torch.compile(), versus 2,619x for the baseline, while lowering DIHARD III DER from 19.09% to 12.73%. At 1.04 seconds, it reaches 865x versus 136x, lowering DER from 19.60% to 13.18%. These are batched throughput numbers, not single-stream end-to-end latency; developers should benchmark the full pipeline on target hardware.

The article distinguishes diarization from ASR: standalone diarization produces speaker activity and timestamps, not words, while ASR produces text without necessarily preserving speaker attribution. A speaker-attributed transcription pipeline combines both. DER itself combines missed speech, false alarms, and speaker confusion, divided by total reference speaker time; overlapping reference speakers each contribute to the denominator. Benchmark settings materially change DER, so the evaluation protocol is part of the result. The Nemotron 3 evaluation contains 901 condition-specific recordings spanning multilingual telephone speech, meetings, near-field and far-field microphones, multi-microphone capture, and difficult acoustic environments, with overlapping speech scored in every evaluation.

Getting started uses NeMo: load nvidia/Nemotron-3-Diarization via SortformerEncLabelModel, set the five streaming parameters in 80 ms frames, call _check_streaming_parameters(), then run diarize() on a 16 kHz mono file. The output is speaker-marked segments like start_seconds end_seconds speaker_id, and intervals need not be mutually exclusive. Combining with offline ASR, the example uses Parakeet TDT 0.6B v3 word timestamps and a midpoint rule to assign each word to the speaker active at its midpoint, marking overlap as ambiguous and leaving unassigned words outside detected speech. The article cautions that this is a simple alignment heuristic, not a separation of overlapping voices, and recommends evaluating word-boundary alignment and both models’ errors before production use.

Deployment considerations include the eight-speaker maximum, sensitivity to noise, reverberation, far-field capture, domain shift, and long conversations. The article advises preserving uncertainty rather than treating assignments as infallible, especially in regulated or safety-relevant workflows. Use is governed by the OpenMDW License Agreement version 1.1. The post also mentions an interactive demo with synthetic conversations, eight-speaker mode, live microphone input, and stress tests, plus ecosystem integrations: Argmax Pro SDK 3 adds real-time speaker attribution with a pre-diarized transcription API, and Baseten and Digital Ocean offer production deployment paths.