(1D) Ordered Tokens Enable Efficient Test-Time Search

Tokenization is a key component of autoregressive generative models, commonly using local tokens (e.g., pixel regions or word pieces) in a fixed order. This work investigates whether token structure affects the ability to steer generation through test-time search, exploring multiple candidates evaluated by a verifier. Using image generation as a testbed, the authors hypothesize that recent 1D ordered tokenizers with a coarse-to-fine structure are more amenable to search than classical 2D grid structures, because intermediate states in coarse-to-fine sequences carry semantic meaning that verifiers can reliably evaluate.

Through controlled experiments, autoregressive models trained on coarse-to-fine ordered tokens exhibit improved test-time scaling behavior compared to grid-based counterparts. Notably, pure test-time search over token sequences—without training an autoregressive model—can perform training-free text-to-image generation when guided by an image-text verifier. The study systematically examines how classical search algorithms (best-of-N, beam search, lookahead search) interact with different token structures, along with the role of verifiers and autoregressive priors.

These results highlight the impact of token structure on inference-time scalability, providing practical guidance for test-time scaling in autoregressive models. The coarse-to-fine ordered tokens enable efficient steering during generation, suggesting that tokenizer design is a critical factor for improving performance at inference time without additional training. This work underscores the potential of ordered tokenization for more effective search-based generation in multimodal AI systems.