Scaling Synthetic Data and Pre-training at Poolside

Synthetic Data Pipeline Strategy

Poolside treats synthetic data as a way to expose implicit rationale and structure in code rather than a replacement for organic data. Their pipeline uses a modular architecture consisting of seeds, metadata, secondary inputs, generator functions, and validators. To prevent model bias and performance degradation, they break complex tasks into simpler, multi-stage workflows. For example, generating a novel involves separate steps for setting, character styles, and plot before generating individual chapters. They also utilize cross-domain porting, such as converting math problems into code, and multi-turn chat pipelines where agents act as judges and evolvers to iterate on task quality. Their internal infrastructure, Hive, allows for configurable agent queues and orchestrators that police generations and dynamically adjust instructions.

Training Verification and Failure Mitigation

To ensure training stability at scale, the team implements strict verification protocols, specifically using hash-based checks on model weights across replicas. If replicas do not return identical hashes, the training run is terminated. This method surfaced several critical issues:

• Broken GPUs: Silent data corruption caused spiky loss curves and massive gradient norms, which were caught by comparing replica hashes.
• Numerical Precision: During tensor parallel accumulation for the LM head, activations grew beyond the capacity of bf16 precision, causing the model to stop converging. Moving this accumulation to fp32 resolved the issue.
• Race Conditions: The use of fp8 kernels led to illegal memory accesses and corrupted gradients, where approximately 0.5% of values were replaced by random noise. This was identified as a blind spot in their hash-checking system, as it occurred in non-redundant training runs.

Model Performance

The team applied these learnings to their 118 billion parameter model, Laguna S. Early evaluations show it outperforming their previous smaller models and competitors like GLM 4.5 Air on coding-specific benchmarks such as MultiPL-E, CodeBench, and SWE-bench Agentless. The team notes that while the model lags in general knowledge benchmarks like MMLU Pro, this is a deliberate trade-off in their data mix to prioritize agentic coding capabilities.