Apple’s system-wide Dictation runs entirely on-device, and the speech it transcribes reaches the foundation model through a tokenizer: an audio encoder that maps short waveform windows to the representation the language model reads. Because that foundation model is sparsely activated under Instruction-Following Pruning, only a small subset of its experts occupies DRAM at any time. The tokenizer, by contrast, is always on, so it competes for the same memory and its parameter count bears directly on power and latency. This paper, by authors affiliated with NVIDIA, Anthropic, and Apple, studies compressing such a tokenizer with distillation.

The core choice is the supervision target. The authors do not distill toward the discrete token or the output distribution. Instead, they train only the student encoder to regress the teacher’s pre-quantizer latent under a squared-error objective on the teacher’s per-frame latent. A single affine layer absorbs the width mismatch between teacher and student. Because the target precedes both the quantizer and the language-model bridge, the same recipe covers both token interfaces the system supports, and it applies both to a tokenizer pretrained alone and to one jointly trained with a language model.

The reported evidence is compact but concrete. At 2.8× compression, the distilled student stays within 1.9% relative WER of its teacher on five of six teacher–student pairs without any fine-tuning. It also improves on an independently trained tokenizer of identical capacity by 3.9% relative. The stated boundary is that this holds for five of the six evaluated pairs; the paper does not claim the same bound for the sixth. The method’s value is that the pre-quantizer latent is shared across token interfaces, so one distillation recipe works in settings that would otherwise require separate treatment.

The source page also lists related readings on on-policy distillation for reasoning models and on a distillation scaling law for compute-optimal allocation between teacher and student, situating this encoder-compression result within a broader distillation research thread.