Selecting AI Models: Beyond the Aggregate Benchmark Index

The Fallacy of Aggregate Indices

Most developers rely on broad leaderboards like the Artificial Analysis Index, which compress diverse capabilities into a single score. This approach is fundamentally flawed for agentic engineering because it ignores the 'three-dimensional' reality of model selection: performance, cost, and speed. A model that ranks high on a general index may be prohibitively expensive or slow for a specific production pipeline, or it may fail to adhere to necessary safety guardrails. The goal is not to find the 'best' model, but to identify the most efficient model for a specific task.

The Five-Benchmark Framework

To make informed decisions, engineers should evaluate models against a curated set of benchmarks that simulate real-world agentic workflows. The speaker proposes five specific benchmarks to replace reliance on generic indices:

1. Terminal-Bench v4.0: The gold standard for pure agentic coding. It uses a real containerized environment and a verifier to check the final state, providing a raw signal on an agent's ability to execute code and solve engineering tasks.
2. APEX Agents: A proxy for non-engineering knowledge work (investment banking, law, consulting). It tests how models handle complex, expert-authored prompts in domains where the agent must navigate high-stakes, document-heavy environments.
3. AutomationBench: Measures performance across business domains (HR, finance, sales) with a critical constraint: the agent must complete the objective without triggering guardrail violations. This is the best measure of 'alignment at the floor level.'
4. AA-Omniscience: A dedicated hallucination benchmark. It distinguishes between correct, incorrect, and 'not attempted' responses, helping engineers identify models that are honest about their limitations—a crucial trait for long-running pipelines where one upstream error poisons the entire chain.
5. DeepSWE v1.1: Focuses on long-horizon software engineering tasks derived from realistic, short prompts. It tests the agent's planning and execution depth over extended timeframes.

Strategic Model Selection

Effective agentic engineering requires moving away from selecting a single 'god model' and toward building a 'model stack.' By analyzing variance across these benchmarks, engineers can find 'alpha'—models that outperform the index in specific areas or provide significantly better cost-to-performance ratios. For example, a model might look identical to a competitor on a headline score but operate at 4x the cost, making the choice obvious once the cost axis is applied. The ultimate metric is 'useful agent output per hour,' which balances token spend, latency, and task completion success.

Notable Quotes

• "Choosing a model is a three-dimensional problem: performance, cost, and speed, together, as one unit."
• "If you could only pick five benchmarks, which five would you pick? This forces decision-making criteria and it forces you to prioritize and push away from these catchall indices."
• "One hallucination upstream poisons every agent downstream in a long-running pipeline."
• "AGI you can't pay for is irrelevant."

Actionable Insights

• Stop using aggregate indices as your primary decision tool. They are proxies of proxies.
• Calculate your own 'useful output per hour.' Don't just look at tokens per task; look at the value delivered per dollar spent.
• Prioritize variance over saturation. If a benchmark shows all models performing at 90%+, it is no longer providing useful information for differentiation.
• Build a model stack. Use different models for different stages of your pipeline based on the specific requirements of that step (e.g., a cheap, fast model for simple tasks, a high-reasoning model for complex planning).