This paper introduces tri-modal masked diffusion models (MDM), described by the authors as the first discrete diffusion model of its kind pretrained from scratch on text, image-text, and audio-text data. That contrasts with earlier work that starts from a unimodal base model and finetunes it for bi-modal generation.

The authors conduct a systematic study of the design space, covering multimodal scaling laws, modality mixing ratios, noise schedules, batch-size effects, and inference sampling defaults. From the batch-size analysis, they derive an SDE-based reparameterization that removes the need to search for an optimal batch size. The key idea is to separate the physical batch size, chosen for compute efficiency such as GPU saturation, FLOP efficiency, and wall-clock time, from a logical batch size that balances gradient variance during optimization. This decouples training-dynamics preservation from hardware constraints.

As a preliminary validation, they pretrain a 3B-parameter model on 6.4T tokens and report strong results on text generation, text-to-image, and text-to-speech tasks. The authors frame this as the largest-scale systematic open study of multimodal discrete diffusion models conducted to date. The provided text is an abstract-level description, so full architecture and evaluation details are not included here.