Building Closed-Loop Evals for Multimodal Agents at Scale

The Agentic Pipeline and Evaluation Strategy

Uber utilizes a multi-stage agentic pipeline to enhance food photography for independent merchants. The system begins with an Image Understanding and Routing Agent that analyzes input images and metadata to decide whether to enhance or skip the asset. If enhancement is required, an Image Editing Agent processes the image, followed by a QA Agent that validates the output against criteria like plating, faithfulness, and color accuracy. If the QA gate fails, the system feeds the rejection feedback back into the prompt generator for up to K iterations. This process is logged in a flat JSON structure to allow for both granular debugging and aggregate performance analysis.

Closed-Loop Autotuning and Diagnosis

To manage model drift and maintain alignment with human preferences, the team implemented a self-correcting loop that operates without human intervention. The system samples production data and compares agent outputs against human-labeled golden datasets. When a mismatch occurs, an Umbrella Diagnosis Agent identifies the specific component responsible for the failure. This triggers a Prompt Optimizer Agent consisting of two sub-agents: a Reflect Agent that analyzes systemic issues in the data and a Synthesize Agent that updates the agent configuration. The new configuration is benchmarked against the golden dataset before being promoted to production.

Guardrails and Failure Mitigation

To prevent reward hacking and hallucinations, the team employs a "Swiss Cheese" model of redundant QA gates. The system tracks a Pass at K metric, measuring the success rate across multiple iterations. Specific failure modes include:

• Faithfulness misses: The agent adds or removes items (e.g., adding shrimp or removing sauce) that were not in the original image.
• Reward hacking: The agent makes superficial changes, such as swapping a plate for a generic bowl, to satisfy the QA gate without improving the image quality.
• Physics and coherence: The agent produces artifacts like plates covering food or unrealistic object placement, which are addressed through coordination with frontier model providers.

Metrics and Production Feedback

Beyond offline evals, the system incorporates online signals including internal dogfooding, merchant feedback, and conversion metrics (add-to-cart rates). These signals are ingested by the diagnostic agent to trigger targeted re-tuning of specific segments, such as specific geographies or dish types, ensuring the system evolves alongside the marketplace.