Lisan al Gaib· @scaling01 · X·· 3 小时前AI 评分40
AI 导读
新的贝叶斯方法很不错 以前，置信区间来自自助法（bootstrapping）。现在，该方法能为每个模型的 ECI 给出一个概率分布 因此你自然就得到可信区间，并能问这样的问题：“模型 A 比模型 B 更强的可能性有多大？” 它还内置了更好的噪声假设： - 噪声在接近 0% 和 100% 时会收缩，因此误差带不再包含小于 0% 或大于 100% 的不可能分数 - 而且每个基准都有各自的噪声水平
正文
the new bayesian methodology is sweet
before, confidence intervals came from bootstrapping. now the method gives you a probability distribution for each model's ECI
so you naturally get credible intervals and can ask questions like: "how likely is it that model A is stronger than model B?"
it also bakes in better noise assumptions:
- noise shrinks near 0% and 100%, so error bands no longer include impossible scores smaller 0% or greater 100%
- and each benchmark gets its own noise level
We’ve updated the Epoch Capabilities Index (ECI) to use a Bayesian methodology. This improves uncertainty estimates and lets us compare models more accurately, even when their individual ECI uncertainty intervals overlap. Frontier model scores are largely unchanged: models stronger than GPT-4 move by less than 1 ECI point on average.在 X 查看被引用的帖子
来源：Lisan al Gaib · x.com