DeepSWE: A Coding Benchmark for Real-World Agent Behavior
Matthew Bermango watch the original →
the gist
DeepSWE is a new software engineering benchmark that uses original, non-public tasks and behavioral prompts to better reflect real-world coding agent performance compared to SWE-bench Pro.
The Breakthrough
DeepSWE provides a more accurate proxy for real-world coding agent performance by utilizing original, contamination-free tasks and behavioral-focused prompts that require agents to explore repositories rather than executing over-specified instructions.
What Actually Worked
- Contamination-Free Task Design: All 113 tasks were written from scratch across 91 repositories in TypeScript, Go, Python, JavaScript, and Rust, ensuring models have not seen the solutions in their pre-training data.
- Behavioral Prompting: Prompts are roughly half the length of those in SWE-bench Pro and focus on desired application behavior rather than prescriptive implementation steps, forcing the agent to discover where and how to apply changes.
- Improved Verification Harness: The benchmark uses a custom verifier that reduces false positive rates to 0.3% (compared to 8.5% in SWE-bench Pro) and false negative rates to 1.1% (compared to 24% in SWE-bench Pro).
- End-to-End Evaluation: The benchmark utilizes the Mini-Suite Agent harness to hold scaffolding constant, allowing for a direct comparison of model capabilities across cost, latency, and token efficiency.
Before / After
- False Positive Rate: SWE-bench Pro (8.5%) vs. DeepSWE (0.3%).
- False Negative Rate: SWE-bench Pro (24%) vs. DeepSWE (1.1%).
- Performance Spread: DeepSWE shows a significant performance gap between models (e.g., GPT 5.5 at 70% vs. Claude Haiku 4.5 at 0%), whereas SWE-bench Pro scores are more tightly clustered.
Context
Existing benchmarks like SWE-bench Pro often rely on public GitHub issues and commits, leading to data contamination where models may have already encountered the solutions during training. Furthermore, these benchmarks often use overly verbose, prescriptive prompts that do not reflect how developers actually interact with AI agents. DeepSWE aims to solve this by testing problem-solving and exploration capabilities through original tasks and a more reliable verification system.
Notable Quotes
- "DeepSWE is a long horizon software engineering benchmark that delivers four major advances over today's public benchmarks."
- "It is a much better test of if the model is writing good code or not, not how well you can explain the problem to the model."
Content References
- tool: SWE-bench Pro, context: cited
- tool: Mini-Suite Agent, context: cited
- tool: HeyGen, context: mentioned