Categorical flow matching (CFM) has recently shown promise for discrete text generation, using a continuous flow between Gaussian noise and one-hot encoded data to enable accelerated sampling.

However, previous work was limited to models under 1B parameters, leaving scalability unaddressed. This paper trains a 1.7B-parameter base flow model on 2.

1 trillion tokens, then self-distills it into a categorical flow map that generates high-quality, diverse text in as few as 4 sampling steps while preserving near-data-level token entropy.

The authors introduce a likelihood bound for CFMs in the semi-discrete setting and demonstrate that the model can score standard language modeling benchmarks comparably to discrete diffusion methods.

They also document practical challenges encountered at scale, offering prescriptive guidance on loss weighting and time scheduling. The key contributions are: scaling CFM to 1.7B parameters and 2.

1T tokens; achieving competitive text generation in 4 steps via self-distillation; deriving a likelihood bound for semi-discrete CFMs; and providing empirical insights into training dynamics for large-scale categorical flow models.