Teaching AI to Find Real Vulnerabilities
AI Engineergo watch the original →
the gist
To train AI for cybersecurity, move beyond simple crash-based benchmarks toward multi-stage, open-world environments that reward chaining exploits and require deterministic grading rather than LLM-as-a-judge.
The Ladder of Exploitation
David Brumley argues that teaching AI to hack requires the same pedagogical approach used for human security researchers: a graduated ladder of tasks. This starts with simple crash-triggering and progresses through memory corruption primitives (arbitrary read/write) to full arbitrary code execution. The primary failure mode in current AI security research is 'benchmaxxing'—creating synthetic, single-vulnerability environments that encourage models to reward-hack the easiest path rather than developing deep reasoning capabilities.
The Failure of LLM-as-a-Judge
Brumley emphasizes that using an LLM to grade another LLM's exploit success is fundamentally flawed. Because models are prone to hallucination and bias, they will often claim success where none exists. Instead, he advocates for deterministic grading oracles. These systems use sandboxed, containerized environments where the grader verifies the exploit by observing actual program behavior (e.g., a crash or a specific memory state change). This ensures that the model's 'proof of vulnerability' (POV) is grounded in reality rather than linguistic confidence.
Moving to Open-World Evaluation
Most existing benchmarks fail because they assume a target has only one vulnerability. In real-world software, targets often contain multiple bugs, some of which are unknown. Brumley proposes an 'audit task' framework where the goal is to find all vulnerabilities. By using stack back-traces to unify crash reports, the system can distinguish between independent bugs. This allows for the calculation of precision and recall: rewarding the model for finding known bugs while penalizing it for 'spamming' invalid exploits or failing to find known issues. This approach prevents the model from getting stuck on the easiest bug and allows for the discovery of zero-days.
High-Value Targets: The V8 Case Study
To test the limits of current frontier models, Brumley's team evaluated them against 41 real-world vulnerabilities in V8 (the JavaScript engine in Chrome). The experiment measured progress across a 16-step capability ladder, moving from in-sandbox crashes to out-of-sandbox arbitrary code execution. While top-tier models achieved 95% success in triggering basic crashes, the difficulty curve steepens significantly when requiring chained exploits to escape the sandbox. This granular measurement allows researchers to identify exactly where models fail, providing a clear trajectory for future training.