AI Agent Maintenance: Why Less Is More
Nate B Jonesgo watch the original →
the gist
Building AI agents is easy, but maintaining them is hard because they break when models improve or source data drifts. The most effective agents are those with a lean, well-maintained 'harness' that is pruned as the model gains capability.
The Case for Pruning
Building an agent is often treated as an additive process where developers pile on tools, memory, and integrations. Vercel demonstrated that agents can actually improve by deleting 80% of their tools. As models become more capable at reasoning and tool use, a complex harness that was necessary for an unreliable model becomes a source of drag and confusion for a stronger one. The goal is not to build the most muscled-up agent, but to maintain a lean, functional workbench that matches the current model's intelligence.
The Two-Way Breakage Problem
AI agents are unique because they break in two directions: when the world around them changes (data drift) and when the model inside them evolves. Unlike traditional software that breaks when it gets worse, agents can become dangerous when they get better. A model that previously required strict guardrails to prevent hallucinations may now be capable of taking 20 plausible but incorrect actions in minutes. If the harness is not updated to reflect these new capabilities, the agent will overreach, producing convincing but incorrect work based on stale documentation or outdated process definitions.
The Five-Point Maintenance Checklist
To keep an agent healthy, operators should treat the system like a sailboat in motion rather than a static application. Regularly audit the following:
- Input Sources: Verify that the data the agent reads is current and that the underlying business processes have not shifted.
- Tool Reach: Audit permissions to ensure the agent has the correct level of access. A permission that was safe for a weak model may be too broad for a stronger one.
- Job Definition: Explicitly define the agent's role. If the model is now capable of planning rather than just summarizing, update the instructions to reflect that shift rather than letting the agent's behavior drift silently.
- Proof Requirements: Require the agent to provide a linkable trail of evidence for its claims, such as specific ticket IDs or source documents, so a human can verify the output.
- Value Assessment: Periodically evaluate if the agent is actually saving time or if it is creating a new pile of work that requires human cleanup.