New University of Chicago paper suggests that an LLM's own account of what matters to its decision is a poor guide to what actually varies across its answers.

That LLMs fail to predict the key questions their own answers split on, which only comparing many answers reveals.

When an LLM recommends something, you want its cruxes, the few sub-questions that drove its conclusion. Asking the model upfront, or reading 1 of its answers, is cheap but may miss what really drives it.

They had models forecast 59 March Madness games and 15 Metaculus events many times, then compared pairs of forecasts to find disputed facts. A reranker made that matching cheap, at an estimated 86 times less compute than a local LLM judge.

In March Madness, predicted cruxes shared 13.4% of clusters with the real ones, below the 39.2% a random shuffle gives. A single answer did better but covered only 28.6% of real cruxes, against a 37.8% baseline.

For high-stakes decisions, sample many answers and look where they disagree, instead of trusting the model's own list of what matters.