Malleable Evals for Self-Changing AI Agents

The Breakthrough

Vincent Koc proposes malleable evaluations that treat evals as adaptive agents: define end-state intents, let agents curate test suites from production traces, enable always-on online evals, and loop in telemetry for self-correction.

Evolution from Static to Adaptive Evals

Traditional software uses unit tests, regression suites, CI/CD, and chaos engineering for measurement. AI evals stick to static benchmarks and handcrafted datasets, which fail for malleable agents that rewrite harnesses and adapt behaviors. Production traces show behavioral drift, stale suites miss edge cases, and 20% of real-world issues evade synthetic tests.

Key Techniques for Intent Engineering

• Progressed from prompt engineering (random word bashing) to context engineering (RAG, tool calling for modular testing) to intent engineering, where agents self-optimize toward user goals using cheap, fast tokens and capable models.
• Agents self-curate eval suites from traces: feed production logs back to detect changes in user behavior or queries, updating tests dynamically.
• Run online, always-on evals with agents performing selective testing instead of exhaustive static runs.
• Embed telemetry in agent loops: harnesses monitor errors, costs, and conditions to self-heal and continue execution.

Why Static Evals Fail and Adaptive Ones Matter

OpenClaw demonstrates harnesses that adapt rapidly, outpacing diff reviews. Benchmarks lag because AI apps evolve—agents handle ambiguity, personality, and novel intents differently per user. Evals must become agentic: optimize toward reward signals like user intent, managing the critical 20% edge cases that break products. Koc urges an agentic mindset for evals, akin to auto-optimization tools that tune toward goals.