For most of the world’s languages, language model pre-training operates in a data-constrained regime: models must repeat their training data many times, which degrades generalization. This paper by Paul Jeha and colleagues (DTU, with work done at Apple) studies two remedies. One is aggressive hyperparameter tuning, such as high weight decay, which regularizes the model by shrinking weights to restrict network capacity. The other is mixing in data from a high-resource auxiliary language: a tunable mixing ratio expands the training distribution and diversifies the training signal with new knowledge.

The authors compare these levers systematically across four model scales from 150M to 1.43B parameters, using Arabic as the low-resource target and English as the auxiliary, over roughly 1000 pre-training runs. Three findings emerge. First, mixing yields larger improvements than hyperparameter tuning on both validation loss and downstream task accuracy, and the gap grows with model size. Second, the benefit is quantified: mixing boosts performance by an amount equivalent to 2–3× the unique target data on validation loss, and 2–13× on downstream task accuracy, with the gain scaling steeply with model size.

Third, target-language validation loss systematically underestimates mixing’s value. Mixing works through two channels: it regularizes by diversifying the training signal, and it contributes knowledge that the repeated target corpus cannot supply. Validation loss captures only the first effect.

The authors’ practical recommendations are to mix in a high-resource language, prioritize the mixing ratio over hyperparameter tuning, and transfer hyperparameters from a small proxy model to the target scale via μP, building on prior work on neural network parameterisations that enables optimal global hyperparameters to be found at small scale and transferred to large models.