Categorizing AI Output Errors for Targeted Fixes

The Four Flavors of AI Error

Most AI corrections fail because users treat every error as a prompt failure. Identifying the root cause allows for precise remediation:

• Real Miss (Objective Error): The AI fails to extract data present in the source or hallucinates information not in the source. Fix: Update system instructions to require explicit confirmation when information is missing.
• Preference (Subjective Error): The output is factually correct but violates stylistic or tonal preferences. Fix: Provide writing samples as context to fingerprint your style.
• Carryover (Contextual Error): Information from previous turns in a long chat thread or irrelevant files in a shared folder leaks into the current task. Fix: Start new chat threads for distinct tasks and maintain focused, task-specific folders for desktop agents.
• Variation (Environmental Error): The world changes (e.g., budget updates, requirement shifts) after the AI generates the output but before delivery. Fix: Adjust the business process to feed real-time data into the AI pipeline before final output generation.

Systematic Error Tracking

To stop repeating the same corrections, maintain a corrections.md file that the AI updates automatically. Use a system prompt to instruct the AI to log the date, the original output, your correction, and the error category. If a specific error pattern repeats, the AI should increment a tick mark in the log.

Perform a monthly or weekly audit by prompting the AI to review the corrections.md file. Instruct it to group corrections with more than two tick marks and suggest the smallest possible change to your prompt, skill, or process to resolve the underlying issue. This prevents prompt bloat while ensuring iterative improvement.