Scaling AI Evals via API-First Workflows at DoorDash

Decentralizing Evaluation Workflows

DoorDash transitioned their AI evaluation platform from a rigid engineering-owned harness to a cross-functional system. By prioritizing stable APIs over bespoke UIs, the platform team enabled strategy and operations staff to build custom annotation interfaces using coding agents. This shift allows domain experts to define quality bars and rubrics without waiting for engineering resources, effectively turning evaluation into a self-serve team sport.

The Continuous Quality Loop

The platform operates on a repeatable cycle: capturing traces and sessions, sampling data, annotating via custom interfaces, creating golden datasets, and calibrating LLM-as-a-judge prompts. Engineering provides the underlying telemetry and API primitives, while product managers and operations teams own the specific judge prompts and annotation logic. This flexibility allows different teams to assign ownership based on their internal org structure, which the platform team treats as a feature rather than a problem to be standardized.

Self-Serve Judge Calibration

To reduce friction, the team built a self-serve UI for judge prompt optimization. Users can run calibration loops using various models, such as Gemini, Claude, or OpenAI, to refine their LLM-as-a-judge metrics. The interface provides side-by-side comparisons of original and optimized prompts, which helps stakeholders build trust in the automated evaluation process. This approach significantly increased iteration velocity and reduced per-annotation costs at scale.