Building Multimodal Collaborative Shopping Agents

The Discovery Loop and Information Gain

Most shopping agents fail because they treat the user as having a well-defined intent. Instead, agents should treat the user's input as a "vibe" and focus on closing the articulation gap. The agent must build a working state from past history, personal context, and reference images, while identifying which variables (like real-time inventory) must be refreshed to keep the state valid. Rather than following a rigid checklist, the agent should calculate the expected information gain for each potential question. For example, if a user wants to redo a room, asking for the room dimensions is the highest-value move because it acts as a hard constraint for all downstream product recommendations.

Multimodal Elicitation and Adaptive Response

Text-based elicitation is often inefficient for subjective preferences. Agents should use visual inspiration boards to ground the conversation and establish a common language with the user. By tracking micro-signals like hovers and clicks on these visual elements, the agent updates its confidence model regarding the user's style preferences. Once the agent has sufficient data, it must adapt the response format to the user's specific goal. A policy question warrants a summary, a product comparison requires a trade-off table, and style exploration requires visual imagery. The goal is to ensure the information is not buried and that the user feels confident enough to commit to a purchase.

Evaluating Agent Performance

Robust evaluation requires auto-raters at every stage of the loop. Key metrics include:

• Counterfactual Sensitivity: Flipping parts of a query to ensure that extracted constraints change or remain stable as expected.
• Turn Efficiency: Minimizing the number of interactions required to uncover hidden preferences.
• Format Accuracy: Ensuring the chosen response structure (table, list, or visual) matches the query type.
• Data Fidelity: Verifying that the model captures information accurately without hallucination.