Data and Environment Curation for Post-Training LLMs

The Curation Recipe for Reasoning

Post-training performance is bottlenecked by data quality rather than compute or model architecture. For reasoning tasks, the most effective curation strategy involves generating multiple reasoning traces per question rather than maximizing the number of unique questions. This diversity in reasoning paths significantly improves model performance during fine-tuning. Additionally, stronger teacher models are not always superior for generating training data; smaller, specialized models often provide higher-quality signals for specific domains.

Improving Agent Reliability

Agents often fail during long-horizon tasks due to tool-use errors or hallucinations. To mitigate this, developers should treat RL environments as structured data. In production scenarios, such as credit card compliance, simple fine-tuning often fails due to data imbalance. Instead, the team found success by injecting structural tags into the prompt-response pairs, which forces the model to adhere to compliance formats rather than hallucinating specific numerical values. This approach simultaneously improves compliance metrics, reduces latency, and lowers inference costs by allowing the use of smaller, custom-tuned models.

The Curation Stack

Building a robust post-training pipeline requires a multi-layered infrastructure. This includes:

• Environment Layer: Tools for building, measuring, and versioning RL environments.
• Orchestration Layer: Sandboxes for generating rollouts, including support for long-horizon checkpointing and state snapshots.
• Optimization Layer: Techniques like Japa (LLM-based prompt optimization) for iterative system prompt improvement.
• Data Layer: Systematic tools (like the Curator library) to handle dataset mixing, filtering, and teacher-model generation.