Everything Is a Rollout: Agent Evaluation and Optimization

The Agentic Rollout Paradigm

Agent development is fundamentally a machine learning problem rather than a traditional software engineering task. Because agentic outputs are non-deterministic, developers must move away from static unit tests toward empirical evaluation. The authors propose treating every agent interaction as a "rollout"—a trajectory generated within a sandboxed environment that can be verified, measured, and optimized. This approach mirrors machine learning workflows: environments serve as training data, reward functions act as loss metrics, and iterative feedback loops function as optimizers.

Standardizing Evaluation with Harbor

Harbor provides a common language for specifying agentic environments, allowing developers to define tasks, run them in isolated sandboxes, and verify outcomes programmatically. The framework supports parallel execution, enabling users to run thousands of rollouts simultaneously to gather statistically significant performance data. By standardizing the directory structure for these environments, Harbor ensures interoperability across different agent architectures, models, and verification rubrics.

Practical Applications and Optimization

Beyond simple evaluation, Harbor facilitates "agentic map-reduce" workflows where developers execute large batches of tasks to aggregate insights or automate internal processes. Users can perform the following actions:

• Map-Reduce Analysis: Run agents across thousands of sessions, then use a secondary model to summarize recurring failure patterns into actionable feedback files.
• Reward-Based Optimization: Use the trajectory data and verification rewards to perform Supervised Fine-Tuning (SFT) or reinforcement learning to improve agent performance.
• Auto-Hill Climbing: Implement evolutionary methods or iterative prompt optimization by feeding evaluation results back into the agent configuration.
• Product Benchmarking: Build internal benchmarks (e.g., RampBench) to evaluate how well specific models interact with internal codebases or APIs, allowing for informed cost-performance trade-offs.