Engineering Agentic Systems: Observability, Specs, and Tokenomics

The Case for Agentic Observability

Engineering production-ready agents is often treated as a "vibe-based" activity, but scaling them requires moving to a rigorous engineering framework. The core problem is that developers often run agents blindly, unaware of token costs, system prompt bloat, or the actual efficacy of their planning specs. Observability acts as the control surface, allowing engineers to treat agents as systems rather than black boxes. By streaming events to a centralized server and persisting them to a database, developers can gain visibility into every tool call, token count, and system prompt, enabling a closed-loop feedback system where performance can be measured and improved.

The Trade-off Triangle: Markdown vs. HTML vs. Visual Specs

There is a common debate regarding the best format for agent planning specs. Testing reveals that the "unreasonable effectiveness of HTML" is not just a marketing claim—it provides a more structured, readable format for agents to interpret complex requirements. However, the choice involves a trade-off between performance, speed, and cost. In some cases, Markdown agents may consume more tokens than HTML agents due to variance in reasoning or focus. The key insight is that "more useful tokens" outperform fewer tokens, provided the tokens contribute to the agent's goal. Visual specs, enhanced by models like GPT Image 2, allow agents to process interface mockups directly, which significantly reduces the planning constraint for complex UI-heavy tasks.

Tokenomics and the Agentic Value Chain

True agentic engineering is about tokenomics: using tokens to generate value and then capturing that value. A fleet of agents is only as good as the revenue or utility it produces. Developers must move up the value chain by first establishing observability, then optimizing for "cost-per-intelligence" rather than just "cost-per-token." If a more expensive model or a more token-heavy spec leads to a higher-quality outcome that saves time or generates revenue, the investment is justified. The goal is to build composable systems where agents can be swapped, measured, and refined based on empirical data rather than intuition.

Implementation Strategy

1. Instrument Everything: Stream every event, turn, and tool call to a centralized dashboard.
2. Analyze System Prompts: Regularly inspect the full system prompt, including all loaded skills and context, to identify bloat.
3. Run Comparative Evals: Use race modes to run different spec types (Markdown, HTML, Visual) against the same task to identify which yields the best performance-to-cost ratio.
4. Focus on Product Agents: Transition from engineering-focused agents (terminal tasks) to product-focused agents (e.g., research, analysis, UI generation) that directly impact business outcomes.