Tokenization has historically been light compute compared to the modeling around it, but as models get faster and workloads scale, the CPU-side tokenizer can starve GPUs of data. Hugging Face’s tokenizers v1, now in release candidate on crates.io, targets this problem directly. The goal is to preserve everything about the existing library — the same token IDs, the same API, vocabularies, and merge ranks — while making encoding 3 to 30 times faster than v0.23.

A tokenizer converts text to integer IDs in four stages: normalization, pre-tokenization, model, and post-processing. Most of the v1 work happens in the model stage, where eight of the ten supported model families use byte pair encoding (BPE); WordPiece and Unigram make up the rest. BPE starts from the bytes of a pre-token and repeatedly joins the highest-ranked adjacent pair until no ranked pair remains, with merges never crossing pre-token boundaries.

Several changes compound. The crate split into a workspace: tk-encode holds the required runtime, while tk-serialize, tk-convert, and tk-train link only when an application needs them. The merge working set lives in a caller-owned scratch buffer, so the loop never touches the allocator. The merge loop itself uses an intrusive doubly-linked list in a single preallocated buffer, so a merge updates two indices instead of moving data, and each candidate pair is packed into a 64-bit value with the merge rank in high bits, making comparison branchless. A thread-local word cache maps pre-token bytes to finished token IDs so repeated words skip merging entirely. One shared tokenizer encodes from many threads at once, with each thread drawing scratch buffers and word caches from its own sub-pool rather than queueing on a single lock.

The splitting stage replaced a regex engine with hand-written functions using SIMD instructions. Since a BPE model’s split pattern is fixed at runtime, an equivalent function can be written once per pattern. The bitcannon approach views input bytes as parallel bitstreams, so pre-token boundaries fall out of boolean operations across whole registers, deciding 64 bytes per register operation. This only applies to a handful of grammars covering most byte-level BPE models; tokenizers with other patterns keep the regex path and none of the speed-up, which explains the variance in results.

Benchmark design matters for tokenizer performance. The tokbench repository used consistent rules: every engine runs the same timing loop, vocabulary load is excluded from timing, output IDs are verified against a baseline hash, medians cover only cells every engine ran, each repeat starts a fresh process, and workers pin to eight distinct physical cores. Crucially, the benchmarks encode distinct documents rather than repeatedly encoding one document; repeated single-document tests can be much faster because the entire document already sits in the word cache. The headline results use distinct documents, with a corpus too large to fit in the cache.

Across the ten model families v1 supports, it encodes text 3 to 30 times faster than v0.23 with one thread on an Apple M4 Max — t5-base at the low end, gpt2 at the high end — and scales at 76% of linear across eight workers, all while producing exactly the same token IDs as the released library. The Python bindings wrap the same code but add per-call overhead not included in these measurements.

Installation is the ordinary cargo add tokenizers –pre; training sits behind a default-on feature that pulls a C++ dependency, so encode-only users can disable it. Post-1.0.0 work includes tok-devices, exploring GPU encoding and batch decoding where text and token IDs stay on the device: upload the vocabulary once, calculate output positions in parallel, and gather bytes on the GPU. It would be an optional component for large batches, subject to further prototyping and measurement.