Scaling Clinical Intelligence and Evaluation at Abridge

Evaluation as an Operating System

Because the gap between generating a clinical response and verifying its accuracy is narrow, Abridge treats evaluation as the primary engineering constraint. To maintain high quality without relying on a single, potentially flawed model, they employ a multi-layered judge system. This includes separate judges for clinical safety, adversarial boundaries, and tone. To ensure clinical accuracy, they avoid using a single "golden response" as a reference, as medical notes allow for legitimate stylistic and structural variation. Instead, they use a rubric-based system where two independent physicians draft criteria for a high-quality response, a third adjudicates those into a final rubric, and a fourth performs quality assurance. An LLM judge then performs semantic matching against these rubric elements to score model performance during iterative development.

Cost and Latency Optimization

Operating at a scale of 100 million medical conversations per year requires balancing frontier-level intelligence with strict cost and latency constraints. Rather than running a single large foundation model for an entire clinical note, Abridge decomposes the note into distinct sections, such as the history of present illness and the assessment and plan. They post-train smaller, specialized models for each section, which reduces inference costs and latency while maintaining high quality. This strategy relies on a proprietary data flywheel, allowing them to train models on specific clinical workflows that general-purpose frontier models may not prioritize. For real-time tasks like in-visit order capturing, they implement a gating system where lightweight models monitor the conversation for specific triggers, only invoking heavier, more expensive models when an order is detected and requires formal matching against the electronic health record system.