Auditing and Cleaning AI Harness Bloat

The Breakthrough

The author discovered that AI performance degradation after model upgrades is frequently caused by 'harness bloat'—an accumulation of legacy instructions, overlapping skills, and redundant system prompts—rather than model limitations. By mapping the entire harness and implementing a just-in-time loading strategy, the author achieved consistent delivery of complex tasks that previously failed due to instruction interference.

What Actually Worked

• Map the entire harness: Before deleting anything, create a visual map of all custom instructions, project files, memory, and skills. Assign each control a row to track its purpose, ownership, and evidence of continued utility.
• Implement just-in-time loading: Instead of loading all editorial guides and context at the start of a session, trigger specialist skills only when the specific work phase requires them. This prevents the model from being overwhelmed by irrelevant instructions.
• Consolidate to a single source of truth: Identify duplicate rules (e.g., 15 versions of a citation guide) and merge them into one master file to prevent rule drift and conflicting instructions.
• Convert 'polite' rules to hard constraints: Replace text-based instructions for formatting (like word counts or JSON structure) with schema-based checks or programmatic validations that the model can test against directly.
• Tailor the harness to the product: Recognize that the same model (e.g., Fable 5 or ChatGPT 5.6) behaves differently depending on the interface (API vs. chat interface). Adjust the harness to match the specific delivery requirements of the product environment.

Before / After

• Initial State: 66 reusable skills and 172 instruction-related files, totaling 27,000 description characters against an 8,000-word Codex discovery budget.
• Performance Impact: A 'thicker' harness with extensive method narration caused Fable 5 to fail delivery requirements (breaking JSON and word limits) twice, while a 'compact' setup succeeded three out of three times on the same task.

Context

The author found that as AI models evolve, users often 'patch' performance issues by adding new instructions to their system prompts. Over time, these additions create a complex, bloated 'chassis'—or harness—that obscures the model's capabilities and causes it to struggle with conflicting or outdated directives. The author developed a 'cleaner skill' to audit these assets, identify redundancies, and enforce a more efficient, modular structure for managing AI instructions.

Notable Quotes

• "If you blame the model for everything you will keep adding instructions to solve problems created by instructions not by the model."
• "The depth should arrive when the work needs it not all up front in a way that confuses the model."