Why Agent Observability Requires Custom Infrastructure

The Shift from Uptime to Quality

Traditional observability tools are designed to monitor deterministic code paths for uptime, latency, and 400/500-level errors. Agent observability requires a different approach because LLM-based applications are non-deterministic and produce highly voluminous, semi-structured data. A single agent trace can exceed one gigabyte, with individual spans reaching 20 megabytes. Because these traces contain significant amounts of unstructured text, standard observability stacks cannot effectively index or query the data to answer questions about agent reasoning, brand alignment, or grounding.

Custom Database Architecture

To handle these requirements, Braintrust built a custom database from the ground up to support three specific needs:

• Write-Ahead Logging: Enables instant visibility into agent interactions as they occur in real time.
• Analytical Indexing: Provides fast filtering for high-volume trace data.
• Full-Text Search: Uses a forked version of the Rust-based library Tantivy to allow engineers to query every trace containing specific keywords, a capability not natively supported by traditional observability databases like ClickHouse.

Human-in-the-loop Annotation

Agent observability shifts the persona from system engineers to subject matter experts, such as lawyers, clinicians, or wealth advisers. These users review traces to grade agent performance and provide written justifications for their assessments. These human annotations serve as the primary training signal for automated scoring functions, allowing teams to scale the identification of failure modes. Once a trace is captured, it can be added to an offline dataset to facilitate rapid iteration and experimentation, effectively bridging the gap between production monitoring and evaluation.