Maturity Phases of Agent Evaluation

The Evaluation Flywheel

Evaluation is not about exhaustive unit testing, which is infinite and counterproductive. Instead, teams should focus on enumerating specific, known failure modes and building tests around those. The goal is to create a flywheel where production traces are captured, analyzed for failures, and fed back into an offline experimentation environment. This allows developers to rerun production workloads as evals, ensuring that each tweak to the agent provides a measurable improvement rather than just a change in behavior.

Maturity Stages of Agent Evals

Teams typically progress through four stages of maturity as their agents grow in complexity:

• Vibe Checking with Justification: Start by having subject matter experts review agent outputs. Crucially, require annotators to provide a written justification for every thumbs-up or thumbs-down. This extracts domain-specific knowledge that is necessary for later automation.
• Scaling with LLM-as-Judge: Use the collected human justifications to train or prompt an LLM to act as a judge. This automates the evaluation of production traces at scale. Always evaluate the judge itself against a ground-truth dataset to ensure it remains aligned with human expectations.
• Deterministic Guardrails: Supplement LLM judges with code-based checks for objective failure modes. For example, trigger a failure if an agent exceeds a specific token count or makes an excessive number of tool calls.
• Handling External System State: Managing CRUD-based tool calls requires representing the state of external systems as they existed when the original trace occurred. Two effective approaches include injecting captured system state directly into the trace or performing timestamp-based queries against vector databases to retrieve the state at the exact moment of execution.

Context

Many teams struggle to move from proof-of-concept to production because they lack a systematic way to maintain confidence in their agents. By treating evals as a method to rerun production rather than a static test suite, teams can play offense, using data to guide development and mitigate risks related to cost, reputation, and compliance.