Building Reliable Agent Harnesses

The Breakthrough

Reliability in agentic systems is a harness problem, not a model problem; failures occur when the system lacks a durable, verifiable chain of custody for state transitions and actions, leading to "silent successes" where the model remains coherent over a corrupted or incomplete history.

What Actually Worked

• Implement a Single Writer Lane: Prevent race conditions by ensuring each mutable state boundary has exactly one ordered commit path, using mutexes or transactions to avoid the "last writer wins" erasure of concurrent operations.
• Enforce State Ownership: Assign every piece of data (transcript, memory, tool result) to a specific system of record. If no owner can replay the fact, the system does not reliably remember it.
• Bind Authority to Actions: Treat approvals as scoped objects that include the actor, session ID, tool arguments, and expiration timestamps. Do not treat approval as a vague, persistent memory that can be retried indefinitely.
• Generate Run Receipts: Move beyond simple transcripts by creating a "run receipt" that records the trigger, inherited state, authority used, execution result, and confirmation of user-visible evidence. This receipt serves as the audit trail for what the system actually allowed and confirmed.
• Apply Deadlines and Watchdogs: Prevent "dangling" tool calls by assigning every external boundary a terminal state (success, failure, timeout, or cancellation) to ensure the system does not queue new work behind a stuck process.

The Run Receipt Audit

To diagnose production incidents, audit a specific trace by answering these five questions:

1. What woke it up? (e.g., user message, webhook, timer, tool result).
2. What state did it inherit? (e.g., transcript, session state, memory snapshot, policy version).
3. Which authority did it use? (e.g., actor, session, tool run arguments, scope, lifetime).
4. What executed? (e.g., tool or API call, attempt number, idempotency key).
5. What evidence survived? (e.g., ticket update, message render, file change, calendar event).

Context

Agentic systems are increasingly complex, involving probabilistic planners that rebuild context for every turn. Because models are stateless, they rely on the harness to assemble the working set. When the harness fails to persist state correctly or allows overlapping writes, the model continues to generate fluent, confident responses based on a broken history. This creates a dangerous illusion of success where the user sees a result, but the system of record remains inconsistent. The goal is to move from a model that merely proposes to a harness that commits and a receipt that proves the outcome.