How to Build and Use Agentic Evals
AI Engineergo watch the original →
the gist
Benchmark numbers are often misleading, but they are essential for iterative improvement. The key is to build custom harnesses, parallelize execution, and perform failure analysis to identify specific levers—rather than just chasing aggregate scores.
The Philosophy of Evaluation
Evaluation is both an engineering and a philosophy problem. Most public benchmarks are either outdated, prone to overfitting, or fail to capture the nuance of agentic workflows that involve multi-step reasoning, file manipulation, and environment setup. Rather than treating benchmark scores as absolute truth, developers should treat them as a signal for hill climbing. The goal is to balance these metrics with real-world user experience (vibes) to ensure the agent is actually useful.
Practical Implementation and Hill Climbing
To move from a low baseline to a high-performing agent, developers must move beyond aggregate scores and perform granular failure analysis. After running a suite of tasks, use an agent to parse the failure traces and categorize them. This allows you to allocate resources toward the specific levers that improve performance.
- Environment Isolation: Use containerized environments to ensure each task runs in a clean state. Tools like Harbor allow for parallel execution of tasks on infrastructure like Modal, which reduces the total evaluation time to the duration of the single slowest task.
- Portfolio Allocation of Failures: Instead of looking at the total score, run a secondary agent over the failure logs to identify why specific tasks failed, such as read errors, test execution failures, or environment setup issues.
- Zone-Based Optimization:
- Zone 1 (Obvious Bugs): Fix infrastructure issues like rate limits, container crashes, or timeout settings.
- Zone 2 (Nuance Improvements): Apply model-specific prompt engineering. Techniques that work for Anthropic models often fail for Gemini or Codex, and these differences explain why a model might perform well in general but fail in a specific harness.
- Zone 3 (Overfitting): Avoid the temptation to cheat the benchmark for the sake of a higher score, as this degrades the actual utility of the agent.
Before / After
- Before: 43% score on Terminal Bench.
- After: Improved performance through container CPU and memory tuning, increased timeouts, and model-specific prompt engineering.