长度不是问题，竞争才是：注意力可预测检索中的上下文退化

Author here. A short summary, since the title is a claim.

Setup: key-value retrieval, 512 to 32K tokens, five open models (0.5B to 3B, four families). At equal length and equal evidence position, we fill the context either with neutral text or with statements in the format of the evidence. On every trial we log the loss on the correct answer and the attention mass on the evidence (a*).

Findings:

Neutral filler is nearly free: 0.97 to 1.00 accuracy up to 32K on Qwen2.5-1.5B (median loss 0.004 nats at 32K).
Competing statements at the same positions: accuracy 0.11 to 0.21 (median loss 4.76 nats at 32K). The damage concentrates on statements that give the queried key a rival value: one such statement multiplies the loss by 9 to 110.
log(1/a*) explains 35 to 58% of the variance in loss across 126,473 trials; log length explains 2 to 8%.
The masked-filler protocol used to argue that length alone hurts (Du et al., 2025) reproduces on MMLU (11 to 18 points lost at 30K). But in retrieval, masking costs 46 accuracy points at 32K while the same filler left visible costs none. So we suggest any length claim include a visible neutral control.

Limitations, stated plainly: the attention result is predictive, not causal (no intervention on a*); one synthetic task; models up to 3B except a 7B in the MMLU replication.

Code: https://github.com/yliusci/length-is-cheap

Data (trial-level): https://huggingface.co/datasets/yliusci/length-is-cheap

Criticism welcome. I'd be especially curious whether people see the same pattern on larger models or real RAG pipelines.

submitted by /u/neom_gluon
[link] [留言]