Cognition FrontierCode Benchmark Analysis

The Shift to Mergeability

FrontierCode moves beyond simple functional correctness by evaluating whether an AI-generated pull request would be accepted by a human maintainer. While traditional benchmarks like SWE-Bench Pro focus on whether code passes a test suite, FrontierCode incorporates blocker criteria, maintainer-defined rubrics, and scope constraints to penalize broad, unidiomatic, or poorly scoped patches that might otherwise pass automated tests.

Evaluation Methodology

Cognition utilizes a multi-layered grading pipeline to reduce false positives and ensure code quality:

• Reverse Classical Grading: The system runs the agent-generated tests against the original base commit. If the tests pass on the buggy code, the agent failed to write a meaningful regression test.
• Adaptive Classical Grading: Using the mute-agent tool, the benchmark adjusts test environments to accommodate valid, non-standard implementation choices, preventing false negatives caused by rigid test expectations.
• Prompt-Based Grading: An LLM reviews the diff against natural language criteria to assess subjective qualities like readability, architectural fit, and idiomatic style.
• Rubric Calibration: Each task undergoes a five-stage pipeline, including "hack reports" where authors attempt to exploit the rubric with bad solutions, and manual audits by both maintainers and Cognition researchers.

Performance and Trade-offs

On the "Diamond" subset (the 50 hardest tasks), Claude Opus 4.8 leads with a 13.4% score and 14.5% pass rate. GPT-5.5 follows with a 6.3% score and 7.2% pass rate. While Opus 4.8 achieves higher scores on the most difficult tasks, GPT-5.5 demonstrates higher efficiency, utilizing approximately 4x fewer output tokens to achieve its results. The benchmark highlights that current models still struggle significantly with production-grade code review, with even the top performers solving only a small fraction of the hardest tasks.