r/MachineLearning· /u/ifaposto·· 6 小时前AI 评分18
自蒸馏除 EMA 外还有哪些教师模型？[R]
Teachers other than EMA for distillation ? [R]
AI 导读
有研究者向社区提问：自蒸馏方法（如 Self-Distillation Fine Tuning，SDFT）通常用 EMA 教师模型提供同步且稳定的目标，是否存在其他类型的教师模型，既能随学生模型一同改进，又能保持稳定、防止学生模型漂移。发帖者希望获得相关论文与代码仓库链接。
正文
Hi guys,
Self-distillation methods, see Self-Distillation Fine Tuning SDFT, usually use an EMA (Exponential Moving Average) teacher to get synced and stable targets. I wanted to ask the community for research papers/ works that consider different types of teachers that also hold these properties (they eventually also improve along with student) but are stable (prevent student from drifting).
any links to repos/ papers would be appreciated.
thanks!
submitted by /u/ifaposto
[link] [留言]
来源：r/MachineLearning · reddit.com