Don't Ship Agent Skills Without Evals
AI Engineergo watch the original →
the gist
Agent skills are often deployed without testing, leading to inconsistent performance. Developers should implement lightweight eval harnesses using JSON-based test cases and script-based assertions to validate skill triggering and task completion.
The Breakthrough
Philipp Schmid demonstrates that agent skills can be reliably validated by moving away from manual "vibe checks" toward automated, lightweight eval harnesses that test for both positive triggering and negative avoidance.
What Actually Worked
- Structure Skills as Directives: Replace essay-style descriptions with clear, imperative instructions. Use the format "Use Skill if Condition" to minimize token costs and improve trigger accuracy.
- Implement Negative Testing: Include test cases where the agent should explicitly not trigger the skill to prevent over-triggering and context pollution.
- Use Script-Based Validators: Instead of relying solely on LLM-as-a-judge, use Python scripts to perform regex checks or SDK-specific assertions on the agent's output. This approach is cheaper and faster for regression testing.
- Run Ablation Tests: Always evaluate performance with and without the skill enabled. If the model achieves the same performance without the skill, retire it to save on context window costs and maintenance.
- Optimize for Token Efficiency: Remove "no-ops"—instructions that do not change agent behavior—from skill files. Keep skill descriptions under 500 words to reduce the per-invocation cost.
Before / After
- Performance Improvement: By creating 117 test cases for the Gemini interactions API, the team improved code generation accuracy from an unspecified baseline to nearly 90%.
Context
Agent skills are often treated as "black boxes" that are accepted after minimal manual testing. Because foundation models are non-deterministic, it is difficult to distinguish between a poor skill definition and a model's inability to handle a task. By treating skills like code that requires unit tests, developers can ensure that specific domain knowledge—such as internal company workflows or new API patterns—is consistently applied without degrading performance over time.
Notable Quotes
- "You wouldn't merge code without tests—so why are we shipping skills without evals?"
- "Skills are not there to live forever. Models get better, behaviors change, expectations change, and the environment changes."
Content References
- Tool: SkillsBench, cited as a benchmark for evaluating agent skills.
- Other: Matt's repository on killing no-ops, recommended for optimizing skill instructions.