AI Agents as Distributed Systems

The Breakthrough

AI agents that interact with external tools are not just model problems but distributed systems problems, requiring deterministic controls to manage the side effects of their probabilistic decision-making.

What Actually Worked

• Implement Idempotency Keys: Every tool call must include a unique request identifier to ensure that retries—triggered by timeouts or network errors—do not result in duplicate side effects like multiple customer refunds.
• Treat Context as Stale State: Manage agent memory as a cache with explicit provenance and invalidation logic to prevent the agent from making decisions based on outdated or conflicting information.
• Enforce Deterministic Guardrails: Constrain the agent loop using circuit breakers, rate limits, and hard ceilings on spend, turns, and parallel calls to prevent runaway retry storms and cascading failures.
• Scope Credentials and Approvals: Replace blanket permissions with granular read/write access and bind human approvals to specific parameters, actors, and expiration timestamps to prevent unauthorized escalation of actions.
• Define Compensating Actions: Design explicit undo or reversal operations for every multi-step transaction to ensure the system can recover to a consistent state when a partial failure occurs across system boundaries.

Context

As AI agents transition from simple text-in-text-out chatbots to systems capable of calling external APIs and modifying databases, they inherit the failure modes of distributed systems. Because agents are probabilistic coordinators, they lack the inherent determinism of traditional workflows. Without robust infrastructure to handle network uncertainty, stale state, and resource exhaustion, these agents risk causing production incidents, such as duplicate transactions or unauthorized database modifications.

Notable Quotes

• "A timeout has never meant failure, it means unknown."
• "The moment a model started calling external services it stopped being a model problem and became a distributed systems problem."
• "When building AI agents we should also ask what the system lets it do when it is wrong."