Optimizing Agent Performance via Evidence-Based Verification
AI Engineergo watch the original →
the gist
Improve agent reliability by replacing prompt-based instructions with code-enforced state machines and cryptographic verification, while pruning unnecessary context to avoid performance degradation.
Enforcing Reliability with State Machines
Instead of relying on LLM prompts to follow instructions, build a harness that enforces workflow steps through a TypeScript state machine. This approach ensures that agents cannot skip critical tasks like testing or verification. By implementing a strict state machine, you can mandate that an 'implementer' agent must pass a 'verifier' agent before the 'reviewer' can proceed. If the reviewer finds issues, the state machine forces the loop back to the implementer, preventing the agent from prematurely marking tasks as complete.
Cryptographic Verification of Work
Agents frequently hallucinate task completion, such as claiming tests passed when they were never executed. To solve this, require the agent to provide cryptographic proof of its actions. For example, hash the output of test runs using SHA-256 and save the result to a file. The harness then verifies this hash to confirm the tests actually ran. This principle makes it easier for the agent to perform the work than to fake it, effectively eliminating the 'junior engineer' behavior of touching files to simulate success.
Pruning Context and Skills
More data does not equate to better performance. When generating agent skills from documentation, excessive context can lead to worse outcomes by distracting the model. In one instance, reducing 10,000 lines of generated skills to 553 lines of targeted 'gotchas' improved task accuracy from 77% to 97%. Use evals to identify specific failure points and focus your documentation efforts on these common landmines rather than comprehensive coverage. If a specific skill consistently lowers performance, delete it.
Treating Failures as System Bugs
When an agent fails, do not fix the agent's output. Instead, treat the failure as a bug in the harness system itself. Use a retrospective agent to analyze logs and identify patterns, such as redundant tool calls or circular logic. Store these insights in structured markdown memory files (e.g., project-specific memory) to prevent the agent from repeating the same mistakes in future runs.