Scaling AI to Long Horizons: Algorithms, Compute, and Mindset

AI Engineer2026-07-31go watch the original →

the gist

Scaling AI to long-horizon tasks requires moving beyond simple context windows to reinforcement learning techniques like value-model bootstrapping, pipeline RL, and high-fidelity environment simulation.

The Shift to Long-Horizon Reasoning

Scaling AI to long-horizon tasks, such as scientific research or complex financial modeling, requires treating the problem as a mindset rather than a simple engineering constraint. Current frontier models struggle with long horizons because they are optimized for procedural, short-term tasks rather than the open-ended, uncertain nature of the real world. To achieve coherence over extended periods, developers must move beyond context window expansion and focus on deliberate token allocation and patient signal extraction.

Technical Strategies for Long Rollouts

To manage the variance and credit assignment problems inherent in long trajectories, the speakers advocate for several specific reinforcement learning techniques:

Value Models for Variance Reduction: Implementing a value model (critic) alongside the policy model helps reduce gradient variance and enables bootstrapping, allowing the agent to receive signal before the end of a long episode.
Pipeline RL for GPU Utilization: To avoid GPU idle time during long inference rollouts, developers should use pipeline RL. This allows training to begin while sequences are still being generated, accepting a controlled amount of off-policy staleness (typically up to eight steps) to maximize hardware throughput.
Compaction and External Tools: When tasks exceed context limits, agents should use compaction (summarizing segments of the trajectory) and external scratch-pad tools like file systems or self-search mechanisms to maintain state without relying solely on the active context window.
Environment Simulation: The authors emphasize that current benchmarks fail to capture real-world complexity. They recommend using high-fidelity environments, such as those hosted on openreward.ai, to better simulate multi-agent dynamics and uncertainty.

The Kelly Bench Experiment

General Reasoning developed the Kelly Bench to test agent performance on long-horizon financial tasks, specifically trading football matches over a one-year horizon. Despite the potential for significant real-world impact, all tested frontier models lost money. This failure highlights the current bias toward coding-centric tasks and the lack of sufficient real-world simulation in existing evaluation frameworks.