Data Quality as a Compute Multiplier

The Data Refinery Approach

Data quality serves as a compute multiplier by steepening the learning curve, effectively allowing a model to achieve higher performance with less compute. Instead of treating data as a firehose, the process should function as an oil refinery that cleans, curates, creates, and composes data to maximize the marginal information gain per token. The primary goal is to identify and prioritize data that is relevant to specific target tasks, as no single dataset is optimal for all use cases.

Curation Techniques

• Rigorous Decontamination: Perform benchmark decontamination using low n-gram thresholds to ensure training data does not leak into evaluation sets, which is essential for interpreting model performance accurately.
• Redundancy Reduction: Remove semantically similar documents that convey identical information, even if the raw text differs, to prevent the model from wasting capacity on redundant signals.
• Synthetic Data Rephrasing: Use high-quality source documents to generate diverse synthetic formats, such as true-false questions, to increase data diversity without risking model collapse. Rephrasing only high-quality documents yields better results than indiscriminate synthetic generation.
• Synergistic Sequencing: Treat pre-training, mid-training, and post-training as a continuous pipeline. Applying domain-specific curation during mid-training can make subsequent post-training phases two to three times more effective by providing a more accurate policy starting point.

Performance Impact

• Efficiency Gains: Curated data allows models to match the performance of larger counterparts (e.g., Qwen 3.5 4B) while using 145x less training compute.
• Multilingual Scaling: Curating English data provides cross-lingual benefits, improving performance in related languages even when non-English training tokens are limited to as few as 6 billion.
• Cost Reduction: High-performance models can be trained for total costs under $20 million, including all R&D and compute, by focusing on data quality rather than raw token volume.

Before / After

• VLM Performance: Curated data yielded a 14 absolute percentage point improvement on vision-language model benchmarks compared to baseline datasets.
• Legal Domain Mid-training: Continued pre-training on 100 billion tokens of domain-specific data improved legal reasoning capabilities by 5 percentage points without causing catastrophic forgetting of general capabilities.
• Post-training Efficiency: Applying post-training to a mid-trained model tripled the performance gains compared to applying the same post-training to a standard instruction-tuned model.