论文提出 Soft-MTP：一行代码改进多 Token 预测（MTP）

Multi-Token Prediction Done Right [R]

AI 导读

一项新论文指出，标准多 Token 预测（MTP）的梯度中混有损害下一 Token 预测的噪声残差，并提出 Soft-MTP 方法将其分离。该方法只需一行代码改动 h_ntp += h_mtp.detach() - h_mtp，即可提取出用于改进投机解码的 KL 自蒸馏信号。作者称 Soft-MTP 在多项任务上提升 MTP 表现，并已开源代码、论文与 HuggingFace 资源。

正文

Sharing one of our latest papers on Multi-Token Prediction (MTP). MTP is now a standard recipe for frontier pre-training but turns out, it can be done much better.

We show the standard MTP gradient hides a KL self-distillation signal (which we want for improving speculative decoding) behind a noisy residual (that harms next-token prediction).

I think the coolest part was the KL gradient–recovery trick to extract the KL gradient. You just need a one-line code change:

h_ntp += h_mtp.detach() - h_mtp

Soft Multi-Token Prediction (Soft-MTP) is essentially a free lunch that improves MTP across a range of tasks.

I'm super excited about this work. Please do check it out below:

💻 Code: https://github.com/JaydenTeoh/Soft-MTP
📝 Paper: https://arxiv.org/abs/2610.38331
🤗 HuggingFace: https://huggingface.co/collections/JaydenTeoh/soft-multi-token-prediction

submitted by /u/jayden_teoh_
[link] [留言]