用 EMA 教师做大规模自蒸馏：ZeRO-3 与 PEFT 的 LoRA 适配器方案
Self-distillation with an EMA teacher at scale: ZeRO-3 + PEFT [P]
作者发布 sdft_zero3 仓库，用 LoRA 微调解决 EMA 教师自蒸馏在大模型上的显存与通信开销问题：骨干网络共享并冻结，只维护学生和教师两个轻量适配器。仓库支持在 ZeRO-3 下运行 SDFT、教师/学生前向时切换适配器、保存与恢复 EMA 教师 checkpoint，并附带修复显存尖峰的 deepspeed 配置，已在 Qwen3.5 和 ToolAlpaca 上测试。
Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc.
We can mitigate this with LoRA finetuning: the backbone is shared & frozen, and we need to maintain two light adapters: one for the student (trained) and one for the teacher (EMA of the student adapter).
Getting this to work cleanly under TRL and ZeRO-3 took some fixes, so I packaged it as a plug-and-play repo. It supports:
-> robust SDFT implementation that runs under ZeRO-3
-> ZeRO-3 + LoRA integration with adapter switching for teacher/student passes
-> Saving and resuming EMA teacher checkpoint
-> working deepspeed configs that fix memory spikes when GPU memory is tight
I tested on Qwen3.5 and ToolAlpaca.
Repo: https://github.com/ifiaposto/sdft_zero3
Feedback/ more examples/ issues welcome!
submitted by /u/ifaposto
[link] [留言]
来源：r/MachineLearning · reddit.com