Nemotron-Labs Diffusion: Fast Text Generation via Self-Speculation
Autoregressive LLMs generate one token at a time, forcing a full model pass per token and loading every weight from memory before any computation begins. This memory-bound bottleneck leaves GPU utilization low, especially at small batch sizes or in latency-sensitive applications. Once a token is emitted, it is final—errors propagate forward with no path for revision. Nemotron-Labs Diffusion tackles this tension by introducing diffusion language models that generate multiple tokens in parallel and iteratively refine them, breaking the rigid token-by-token dependency.
The model family (3B, 8B, and 14B scales, plus an 8B VLM) supports three inference modes from the same checkpoint: standard autoregressive for backward compatibility, pure diffusion mode that fills 32-token blocks via iterative denoising, and self-speculation where diffusion drafts candidate tokens and autoregressive verification confirms them losslessly at temperature zero. Training started from a pretrained AR model using a joint AR+diffusion objective on 1.3T tokens, then fine-tuned on 45B tokens. The 8B model shows 1.2% improved average accuracy over Qwen3 8B, while self-speculation reaches up to 6.4× higher tokens-per-forward-pass compared to standard AR decoding. Deployment is integrated into SGLang, with the inference mode selected via a single config toggle.
For builders, the key takeaway is that diffusion and autoregression no longer need to be separate model families—Nemotron-Labs Diffusion collapses them into a single model with a deployment-time choice. The self-speculation mode is the most practical win: it combines the throughput of parallel drafting with the correctness guarantees of causal verification, delivering roughly 4× speedup on B200 hardware without accuracy loss. This architecture also gives developers a built-in inference budget control by reducing refinement steps. The models, training code via Megatron Bridge, and HuggingFace checkpoints are all available under commercially friendly licenses, lowering the barrier for production experimentation.
