Building an AI Oracle for Multi-Model Adjudication
the gist
The author developed an 'Oracle' tool that automates multi-model consensus by prompting independent models for unbiased feedback, allowing the primary agent to synthesize diverse perspectives and correct its own reasoning errors.
The Oracle Pattern for Model Adjudication
The author addresses the common bottleneck in agentic workflows where an LLM asks the user to choose between two potential paths, often leading to indecision or biased feedback loops. Instead of manually copying and pasting outputs between models, the author built an 'Oracle' tool that automates the adjudication process. When triggered, the primary model constructs an unbiased summary of the decision point and queries secondary models (such as Grok or CodeX) to provide independent evaluations. This forces the system to surface conflicting viewpoints and factual gaps that the primary model initially overlooked.
Implementation and Workflow
To avoid the bias introduced by feeding one model's output directly into another, the system follows a specific orchestration sequence:
- Unbiased Prompt Construction: The primary model generates a neutral description of the decision problem, stripping away its own previous reasoning to ensure secondary models evaluate the raw task.
- Parallel Querying: The orchestrator sends this neutral prompt to multiple distinct models, capturing their unique training biases and factual knowledge bases.
- Synthesis and Verification: The primary model aggregates these external opinions. Each secondary model provides its reasoning, which is then presented to the user alongside the primary model's original plan.
- Iterative Correction: The primary model is designed to acknowledge when secondary models identify missing information or superior alternatives, often resulting in the primary model explicitly changing its initial recommendation based on the Oracle's feedback.
Impact on Reasoning
This approach shifts the role of the primary model from a solitary decision-maker to an orchestrator of a multi-model debate. By surfacing external citations and alternative naming conventions or logic that the primary model missed, the system significantly reduces the frequency of 'hallucinated' confidence. The author notes that in more than 50% of cases, the primary model updates its stance after receiving feedback from the Oracle, demonstrating that the value lies not in one model being 'better' than another, but in the diversity of their outputs.