数据分布漂移时如何处理 split conformal prediction 的覆盖率保证？

How do you handle coverage guarantees from split conformal prediction when the data shifts? [D]

AI 导读

针对 split conformal prediction 在数据不再可交换时覆盖率保证失效的问题，有实践者提出三种应对思路：定期重新校准、监控经验覆盖率并在下降时告警，或改用自适应变体。对于预测集包含多个标签的样本，讨论还涉及将其路由给人工、LLM 或回退规则。

正文

Split conformal prediction gives you a clean statement: with a calibration set exchangeable with future data, your prediction sets cover the true label at the rate you chose. The trouble is that production data rarely stays exchangeable. Categories get added, ticket wording changes, and a calibration set from three months ago quietly stops being representative.

I’m interested in how people deal with this in practice. Do you recalibrate on a schedule, monitor empirical coverage and alert when it drops, or use an adaptive variant? And how do you decide what to do with the cases where the set contains several labels, such as route them to a human, an LLM, or a fallback rule?

submitted by /u/camerongreen95
[link] [留言]