Moving Beyond Chatbot Evals for Production Agents
AI Engineergo watch the original →
the gist
Stop relying on static chatbot-era benchmarks. Instead, treat agent evaluations as code, prioritize detecting when issues started and their impact size, and use agents to investigate anomalies rather than detect them.
Shift from Benchmarks to Code-Based Evals
Most public evaluation discourse remains stuck in the chatbot era, relying on static datasets that break whenever models or tool harnesses change. Instead of investing months in brittle benchmarks, treat agent evaluations as standard software tests. Use frameworks like vitest or OpenAI's macro evals to run tests locally as code. This approach allows for better integration with CI/CD pipelines and ensures that evaluations evolve alongside the agent's logic.
Prioritize Temporal and Impact Analysis
When managing production agents, the most critical metrics are not just identifying errors, but knowing when an issue started and what percentage of users it affects. Without this temporal context, teams waste time chasing noise. Avoid the common pitfall of relying on naive trace clustering, which is useful for one-off analysis but fails to scale or provide actionable boundaries for root cause analysis. Instead, define custom classifiers in a sandbox environment to track specific failure modes at production scale.
Tactical Anomaly Investigation
Agents are inherently poor at anomaly detection. Do not ask an agent to identify anomalies; instead, use deterministic methods like keyword frequency spikes to surface potential issues, then task the agent with investigating those specific, tractable signals. This separation of concerns prevents the agent from hallucinating patterns in logs and focuses its reasoning capabilities on diagnosing known deviations.