Artificial Analysis· @ArtificialAnlys · X·· 3 小时前AI 评分37
AI 导读
决策模型在知识和推理任务上校准很差。对于所选答案的几乎所有概率水平，所有决策模型的准确率都低于该概率。这一差距在 AA-Omniscience-Decision 和 HLE-MultiChoice 上最大，而大多数模型在 WildGuardTest（分类）和 When2Call（工具调用）上校准相对较好。
正文
Decision models are poorly calibrated on knowledge and reasoning tasks. Across almost all probability levels for chosen answers, accuracy falls below that probability for all decision models. This gap is largest on AA-Omniscience-Decision and HLE-MultiChoice, while most models are relatively well-calibrated on WildGuardTest (classification) and When2Call (tool calling)
来源：Artificial Analysis · x.com