Observability and Evaluation for AI Agents

The Breakthrough

Effective observability for non-deterministic AI agents requires shifting from code-based auditing to telemetry-based auditing, using OpenTelemetry traces to reconstruct agent execution paths and identify logical failures like incorrect tool-calling sequences.

What Actually Worked

• Implement OpenTelemetry auto-instrumentation to generate traces and spans, which serve as the audit record for agent behavior.
• Categorize evaluation signals into five distinct flavors: LLM-as-a-judge, human feedback, golden datasets, deterministic logic checks (e.g., JSON schema validation), and business metrics.
• Apply evaluation at varying scopes depending on the complexity of the failure: single-span (input/output of one call), multi-span (data flow across components), trajectory (sequence of tool calls), or session (state machine/user satisfaction).
• Use distributional analysis to compare agent performance across different execution branches, identifying which paths contribute to high latency or regression.
• Automate the observability loop by using AI agents to scan traces, surface errors, and generate relevant evaluations dynamically rather than relying on manual dashboard monitoring.

Context

Debugging AI agents is difficult because the execution path is non-deterministic and changes with every run. Traditional software debugging tools fail to capture the state of these systems. The author advocates for a standardized approach using OpenTelemetry to capture traces, which allows developers to treat AI performance as a data-driven engineering problem rather than a black-box mystery. The goal is to move toward an automated flywheel where the system identifies its own performance issues and generates the necessary tests to fix them.

Content References

• tool: Arize Phoenix, Arize AI, context: recommended
• tool: Arize AX, Arize AI, context: mentioned
• paper: Anthropic Managed Agents paper, Anthropic, context: cited