Fixing AI Clinical Note Errors via Dynamic Evaluation

The Failure of Static Evaluation

Production ambient scribes frequently generate notes that are syntactically correct but clinically dangerous. In a real-world study, one in 20 notes contained errors capable of causing significant harm, while one in five contained important omissions. These errors often persist because static evaluation systems—even those using frontier models and detailed rubrics—cannot distinguish between harmless noise (e.g., a patient's holiday destination) and critical clinical indicators (e.g., a specific travel history suggesting parasitic infection). Because the definition of a 'good' note is tacit, contextual, and constantly shifting, pre-specified rubrics inevitably go stale or fail to account for edge cases.

The Discovery-Capture-Calibrate Loop

Instead of attempting to write a perfect, static rubric, developers should implement a dynamic, three-step evaluation loop that treats 'taste' as a set of evolving examples rather than a fixed specification:

• Discover: Analyze production outputs to cluster failure modes. This creates an ontology of errors based on real-world data rather than whiteboard-based assumptions.
• Capture: Have domain experts provide free-form comments and corrections on real outputs. This raw material serves as the ground truth for what actually matters in a given domain.
• Calibrate: For each new output, retrieve the most relevant historical cases, expert corrections, and guidelines into the judge's context. This allows the evaluator to reason about the current note using the same logic experts applied to similar past cases.

By moving from a static rubric to a system that retrieves case-specific context, the evaluator gains the ability to identify subtle omissions and meaning-flips that traditional automated checks miss. This approach is not limited to healthcare; it applies to any high-stakes domain where being 'confidently wrong' carries significant cost.