Evaluating Web Agent Planning via Fine-Grained Skill Decomposition

The Breakthrough

Surgan Jandial demonstrates that web agent planning can be decomposed into discrete, measurable skills—specifically temporal ordering, future-state prediction, action selection, and error correction—which allows for inexpensive, interpretable performance benchmarking that correlates with overall task success.

What Actually Worked

• Skill Decomposition: The research identifies four core planning sub-skills by analyzing agent traces during web tasks to understand the cognitive steps required to navigate from an initial UI state to a goal state.
• Synthetic Test Generation: Existing datasets are repurposed to create automated, low-cost benchmarks that test these skills in isolation, avoiding the need for human-in-the-loop evaluation or expensive full-trajectory runs.
• Counterfactual Testing: The evaluation uses simple, targeted questions—such as presenting two screenshots and asking which state must occur first to solve a task—to force the model to demonstrate specific reasoning capabilities.
• Performance Filtering: These independent skill scores serve as a preliminary diagnostic tool, allowing developers to filter out underperforming models before committing to costly, complex end-to-end evaluations.

Context

Evaluating web agents typically requires running full, multi-step trajectories that are computationally expensive and difficult to debug due to the "needle in a haystack" nature of identifying exactly where a model failed. By isolating planning into sub-skills, this approach provides a diagnostic layer that identifies specific weaknesses in a model's reasoning process. The results indicate that most open-source models currently perform below 50% on these fundamental planning tests, highlighting a significant gap in current agentic capabilities.

Content References

• paper: On the Fine-Grained Planning Abilities of Web Agents, Surgan Jandial et al., EMNLP, cited.