TokenOps: A Control Plane for AI Agent Cost Governance

The Breakthrough

The authors developed an out-of-band control plane that moves AI cost management from simple request-level throttling to run-time steering, allowing agents to adjust their behavior dynamically to stay within budget without prematurely killing the process.

What Actually Worked

• Boundary Annotations: Developers annotate existing methods with a @boundary decorator, which tracks input/output telemetry and provides a channel for the control plane to inject instructions back into the agent runtime.
• Governor Configuration: A governor instance is initialized with specific allowed actions, ensuring the control plane can only apply authorized modifications (e.g., truncating tool outputs or modifying system prompts) to the agent.
• Steering vs. Halting: Instead of only using circuit breakers (Halt), the system employs a 'Cost Guard' that monitors token consumption velocity. When a budget overrun is predicted, it injects instructions to the LLM to prioritize succinctness or summarize outputs.
• Segmented Budgeting: The system supports hierarchical cost tracking by grouping runs into segments based on custom dimensions (e.g., user cohorts or project IDs), allowing for fine-grained policy enforcement rather than global caps.

Before / After

• Average Spend: Reduced by 78% across benchmark runs on open-source repositories.
• Completion Rate: Increased from 67% to approximately 96% by replacing hard-kill throttling with adaptive steering.

Context

Existing AI cost management tools typically operate at the model gateway level, offering only binary options like hard budget caps or model routing. This approach often leads to runaway loops or context bloat that kills agent runs unnecessarily. The authors propose an architecture that sits between the code and the model call, providing observability and active intervention capabilities that mirror the control surfaces found in previous SaaS and cloud computing eras.

Notable Quotes

"Simple throttling holds the bill down by killing runs, so Tisha Chawla and Susheem Koul built their control plane to steer instead."

Content References

{"type": "tool", "title": "browser-use", "context": "mentioned"}, {"type": "tool", "title": "MetaGPT", "context": "mentioned"}, {"type": "tool", "title": "LangChain", "context": "mentioned"}