Measuring Agent ROI via Entropy and Verification
AI Engineergo watch the original →
the gist
Agents suffer from a measurement problem where token spend is often decoupled from actual value; the solution is to map tasks on an entropy matrix to identify those where verification is cheaper than execution.
The Measurement Problem
Agents are currently suffering from a "Watt's problem" where early adopters are biased and token spend is often disconnected from real-world outcomes. Because coding agents generate output faster than humans can review it, the bottleneck has shifted from generation to verification. Without a clear way to measure the value of the output, users fall into a cycle of overspending on tokens followed by austerity, rather than treating agents as a predictable tool.
The Entropy Matrix for Task Selection
To determine if a task is suitable for an agent, evaluate it against two axes of uncertainty: the steps required to complete the task and the criteria required to verify the result.
- Low Step Uncertainty: If the path to completion is predictable, do not use an agent. Write a script instead.
- High Step Uncertainty: If the task is out-of-distribution for the model or has sparse rewards, it is likely a poor candidate for current agentic architectures.
- High Criteria Uncertainty: If verifying the output requires as much effort as performing the task, the agent provides no net efficiency gain.
Verification as an NP-Style Problem
The ideal agentic task resides in a sweet spot where the work is complex enough to require an agent, but the verification process is relatively repeatable. This mirrors NP-style problems where it is computationally cheaper to check a solution than to generate one. Once a repeatable verification rubric is established, developers can deploy a secondary agent to handle the validation of the primary agent's output, effectively automating the review bottleneck.