Daily curated AI insights you can't miss.
Marah believes synthetic data does not replace organic data but uncovers hidden structures to supplement training.
Robert explains they ensure training correctness by hashing model replicas for consistency check, crashing training immediately upon inconsistency.
The 118B model surpasses GLM 4.5 Air and DeepSeek V4 Flash Max on coding benchmarks, achieving the strongest performance in agentic evaluations.
Good code data runs out, so poolside manufactures more of it. The team treats synthetic data as a complement to organic data, not a replacement, using it to expose implicit rationale, planning, and structure that organic data often hides. For their XGen-2 model, synthetic data made up 13% of the pre-training mix; the corpus has since grown to 6 trillion tokens.
The synthetic data pipeline is built from six modular components: seeds, primary inputs, metadata, secondary inputs, a generator function (which can be an agent with tools or a prompted model), and supplementary functions like filters and validators. Pipelines span a spectrum from cheap, scalable rephrasing (seed-heavy, using smaller models) to complex orchestrated workflows that break tasks into simpler steps. Multistage pipelines generate a novel one chapter at a time, or port data across domains (e.g., math problems to code). Multi-turn chats involve two agents talking or a judge-evolver loop. An orchestrator (Hive) polices generations, dynamically choosing which agent goes next and enforcing quality.
On the pre-training side, the team trusts nothing. They run multiple replicas of the same model in parallel, periodically compare weight hashes, and crash the training if any hash differs. This catch has revealed several failures invisible at smaller scale. A broken GPU caused silent data corruption and a spiky loss curve; the hash check caught it. A numerical precision bug in tensor parallel accumulation for the unembedding head—defaulting to BF16—caused activations to grow until the model stopped converging. Moving the accumulation to FP32 immediately restored convergence. A race condition in FP8 training kernels (from DeepChem) silently corrupted about 0.5% of gradients, replacing them with random values. This was a blind spot because hash checks don’t compare forward/backward results across replicas. The team has a fix in a public PR.
These insights fed into the design of Laguna S, a 118 billion parameter model (8B active) trained on 30 trillion tokens across 4,000 GPUs. Early base-model evaluations show Laguna S outperforming GLM 4.5 Air, Nemotron 360, and DeepSeek V4 Flash Max on coding benchmarks like LiveCodeBench, BigCodeBench, and SweepBench agentless multilingual. It is competitive on BigBench Hard but trails on general knowledge benchmarks like MMLU Pro—a deliberate trade-off since the focus is agentic coding. The recipe held at scale, and the model will be released as open weights.
Good code data runs out, so poolside manufactures more of it. The team treats synthetic data as a complement to organic data, not a replacement, using it to expose implicit rationale, planning, and structure that organic data often hides. For their XGen-2 model, synthetic data made up 13% of the pre-training mix; the corpus has since grown to 6 trillion tokens.
The synthetic data pipeline is built from six modular components: seeds, primary inputs, metadata, secondary inputs, a generator function (which can be an agent with tools or a prompted model), and supplementary functions like filters and validators. Pipelines span a spectrum from cheap, scalable rephrasing (seed-heavy, using smaller models) to complex orchestrated workflows that break tasks into simpler steps. Multistage pipelines generate a novel one chapter at a time, or port data across domains (e.g., math problems to code). Multi-turn chats involve two agents talking or a judge-evolver loop. An orchestrator (Hive) polices generations, dynamically choosing which agent goes next and enforcing quality.
On the pre-training side, the team trusts nothing. They run multiple replicas of the same model in parallel, periodically compare weight hashes, and crash the training if any hash differs. This catch has revealed several failures invisible at smaller scale. A broken GPU caused silent data corruption and a spiky loss curve; the hash check caught it. A numerical precision bug in tensor parallel accumulation for the unembedding head—defaulting to BF16—caused activations to grow until the model stopped converging. Moving the accumulation to FP32 immediately restored convergence. A race condition in FP8 training kernels (from DeepChem) silently corrupted about 0.5% of gradients, replacing them with random values. This was a blind spot because hash checks don’t compare forward/backward results across replicas. The team has a fix in a public PR.
These insights fed into the design of Laguna S, a 118 billion parameter model (8B active) trained on 30 trillion tokens across 4,000 GPUs. Early base-model evaluations show Laguna S outperforming GLM 4.5 Air, Nemotron 360, and DeepSeek V4 Flash Max on coding benchmarks like LiveCodeBench, BigCodeBench, and SweepBench agentless multilingual. It is competitive on BigBench Hard but trails on general knowledge benchmarks like MMLU Pro—a deliberate trade-off since the focus is agentic coding. The recipe held at scale, and the model will be released as open weights.
