Domain-specialized models inherit a tokenizer built for general-purpose text, and that inheritance is a measurable cost, especially for small models. In a production Portuguese essay grader built on Qwen3-4B, the authors of Dharma-AI‘s upcoming VocabX paper rebuilt the vocabulary around inference cost instead of token frequency and cut measured inference cost by 77% without changing the base model, the training data, or the training recipe.

The grader’s tokenizer was still Qwen3’s original vocabulary of 151,669 entries, built for many languages and kinds of text. That created two kinds of waste: the tokenizer fragmented the JSON envelope, competency names, and evaluation phrases that the grader writes on every request into many tokens, and the model scored all 151,669 vocabulary entries at every generated token even though the task almost never uses most of them. The article explains why counting tokens is not the same as counting cost. Prefill is compute-bound and relatively cheap; decode is memory-bandwidth-bound, and in the measured grader a generated token cost 7.4 times a token read, a ratio that varies with hardware, batch size, and context length. The LM head also holds one row of parameters per vocabulary entry—roughly 10% of Qwen3-4B‘s parameters—and reads the entire matrix at every decode step. In the grader, the output layer shrank from about 152,000 to about 65,000 rows.

Existing vocabulary adaptation methods such as fast vocabulary transfer (FVT), multi-word tokenization (MWT), and Liquid AI’s in-place BPE already reduce fragmentation, but they choose tokens by frequency rather than by cost and work on only a single layer. VocabX weights each saved token by the price of the field it appears in, so a token saved in the model response is worth 7.4 times one saved in the input. It works on both the sub-word layer and a multi-word layer that turns repeated spans such as whole lines and JSON envelopes into single macro tokens, using five miners specialized in long words, whole lines, expressions, complete sentences, and punctuation structures such as JSON and tags; averaged over eight corpora, the last one contributed the most, and on code it alone accounted for 17.5 points. The article also reports that without all five miners, token reduction on the grader would drop by 55.6 points. VocabX controls vocabulary size in two steps: the sub-word merge replaces the base vocabulary with a smaller domain-specific one, and macros enter under a fixed budget. It also handles candidates the sample cannot see: instead of assigning zero gain to expressions absent from sampled documents, it estimates their gain statistically. That change alone raised token reduction on a contracts corpus from 44.7% to 52.8%.

In the grader, seven versions trained with the same base model, data, recipe, and seed differed only in the tokenizer, and all served the same 500 requests on an RTX 5090 with vLLM. The average response dropped from 350 to 101 tokens, measured inference cost dropped 77.0%, and the bench score (grading accuracy combined with screening of off-topic or rule-breaking essays) rose from 0.703 to 0.810. One design decision mattered: fusing each grade with the surrounding JSON into one macro token was the most compact option but lowered the bench score to 0.689; giving the grade its own token cost two extra tokens per response but raised the score to 0.810 and was cheaper in measurement (77.0% versus 73.3% cost reduction). The authors’ lesson is that compression can hurt when the compressed span carries the very decision the model exists to make; the goal is the cheapest vocabulary that preserves the tokens that matter.

Across five public domains (contracts, medical papers, scientific papers, news, Python code) and three model families (Qwen3, Llama-3.1, Gemma-3), VocabX compressed more than the strongest reference method in all fifteen combinations by 1.2 to 9.3 points, and led in estimated inference cost with average margins of 10.2 points on Qwen3, 7.5 on Llama-3.1, and 13.6 on Gemma-3. On contracts, the pruned version reached 62.6% token reduction with 113,426 vocabulary entries, versus 61.8% for the best in-place BPE configuration using 207,803 entries. The article is careful about scope: VocabX does not make the model smarter; it makes the model stop paying piece-by-piece for text it writes repeatedly and stop scoring a vocabulary it will never use. The numbers come from the current version of the paper, still under review, and may change before publication.