LLM Instruction Following Capacity Has Increased Tenfold

The Tenfold Capacity Jump

Laurie Voss replicated the IFScale benchmark, which measures a model's ability to include specific keywords in a generated report, to determine if the previously established 200-instruction limit still holds. While models from a year ago (GPT-4.1, Claude Sonnet 4, Gemini 2.5 Pro) began failing around 200 to 300 instructions, current frontier models (GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, Deepseek V4 Pro) successfully handled 2,000 to 5,000 instructions. This shift effectively eliminates the need for aggressive prompt compression or complex sharding of instructions across multiple agent files.

Failure Modes and Verification

As capacity has increased, failure modes have become more nuanced and harder to detect. Models no longer simply forget instructions; they exhibit distinct, model-specific behaviors when pushed beyond their limits:

• Deepseek V4: Predictably forgets instructions as the count increases.
• Claude Opus 4.7: Triggers safety filters and refuses requests at the API level if instructions contain sensitive-sounding terminology.
• Gemini 3.1 Pro: Exhausts its internal thinking budget on complex instructions, resulting in no output or incomplete responses.
• GPT-5.5: Begins the task but may abandon it halfway through, explicitly stating that the request is unreasonable.

Because these failures can appear as polished, partial outputs, developers can no longer rely on the model's output quality as a proxy for instruction adherence. Verification is now a mandatory step in production, requiring automated checks or secondary LLM-based evals to confirm that all constraints were met.