Task Fidelity Scaling Laws: Data Quality in Agentic Benchmarks

The Impact of Task Fidelity on Model Training

Kobie Crawford from Snorkel demonstrates that task quality is the primary driver of performance gains in agentic RL training. Using terminal-based agentic tasks, the research team compared the training uplift of models fine-tuned on "accepted" versus "rejected" tasks. Accepted tasks were defined by four criteria: achievability, non-triviality, functional correctness, and environment reliability. The study found that high-quality tasks resulted in a 6% improvement in model performance, while low-quality tasks yielded only a 1% improvement, representing a 5x difference in training efficiency for the same compute budget.

Identifying High-Quality Tasks

Accepted tasks consistently exhibited specific behavioral markers that distinguished them from noisy, rejected tasks. The team identified the following characteristics of high-quality agentic tasks:

• Increased Complexity: Accepted tasks required twice as many tool calls and more output tokens, indicating deeper reasoning requirements.
• Meaningful Failure Modes: Failures in accepted tasks were typically due to genuine logical difficulty, whereas rejected tasks often failed due to environmental issues or mismatches between the task definition and the test suite.
• Clear Specifications: Rejected tasks frequently suffered from underspecified goals or implicit dependencies that were not provided in the model context, creating noise rather than challenging the model.
• Expert-in-the-loop Validation: Snorkel utilizes human experts to build and verify tasks, ensuring that the ground truth logic is sound before the data is used for model training.

Before / After

• Low-quality task training: 1% improvement in base model performance.
• High-quality task training: 6% improvement in base model performance.

Context

Snorkel focuses on producing high-quality datasets for foundation models. The experiment was designed to provide empirical evidence for the thesis that data quality is the critical bottleneck in agentic model performance. By containerizing tasks and enforcing strict acceptance criteria, the team isolated the impact of task fidelity from other variables like model architecture or compute budget.

Content References

• Tool: TerminalBench — cited as the framework for agentic coding tasks.
• Tool: Swe-bench — mentioned as a benchmark for comparison.