Moving LLM Outputs from Average to Outlier

Forcing Outlier Perspectives

LLMs are trained to provide the most probable, average response to any given prompt. To move beyond this, users must explicitly instruct the model to ignore common consensus and adopt an expert lens. A simple starting point is to request two distinct answers: the most common response and an expert response that highlights non-obvious, actionable insights.

Expert-Led Decision Frameworks

When you possess domain expertise, you can improve output quality by forcing the model to structure its reasoning. Instead of asking for a general opinion, require the model to list six specific factors a high-level practitioner would weigh, then rank those factors based on your specific context. You can further refine this by using an iterative interview process where the model asks you one question at a time to gather necessary context before providing a final recommendation. For complex decisions, force the model to analyze five specific dimensions: real trade-offs, potential downsides, second-order effects, common mistakes, and necessary conditions for success.

Managing Uncertainty and Hallucination

When you lack domain expertise, you cannot easily verify the model's output. To mitigate this, instruct the model to label every claim as either "backed" or "inferred." A "backed" claim must point to a specific line or rule in your provided source material, while an "inferred" claim must be labeled as such, accompanied by a specific question you should verify before acting. Finally, for high-stakes decisions, run the same prompt across two different models (e.g., Claude and GPT-4o). If the models disagree, treat the output as a signal that human judgment is required to resolve the discrepancy.