15M 参数决策模型对比：线性递归 vs Transformer 注意力，精度打平但延迟与训练成本差距明显
I trained a 15M "decision-only" model two ways — linear recurrence vs Transformer attention. Accuracy was a coin flip; latency and training cost weren't.[R]
有人用相同预算和任务训练了两个约 15.09M 参数的"仅决策"模型，对比线性递归骨干与标准 Transformer 注意力。在 2k–8k 字符长上下文检索决策任务上，两者测试精度分别为 26.8% 和 28.2%，接近 25% 随机水平；但 T=8192 时线性模型延迟快 2.37×（108.6 vs 257.7 ms），训练速度差距约 6–7×。
Inspired by TypeSafe's Jev (a non-generative model that only outputs bounded decisions — choices, probabilities, calibrated confidence), I built two small decision models end-to-end on identical budgets and tasks to answer one question: which substrate makes the better Jev?
Repo: https://github.com/stube123890-hue/jev-decision-lab
Setup (V4, the main result): - ~15.09M params each, budget-matched to ±0.02% - jev-v8a: linear recurrent backbone (my Gated Segmented State Space architecture from linear-attention-lab) + decision head - jev-transformer: standard attention backbone + identical head - New task: long-context retrieval-decision — given a 2k–8k character passage, decide which of 4 sentences appears in it (2000 train / 500 val / 500 test) - Trained on a Colab T4, everything reproducible in the notebooks
Results: - Accuracy: noise. v8a 26.8% vs transformer 28.2% on test (chance = 25%). Neither learns much at this scale — reporting the loss straight. - Latency: real crossover. Transformer faster below ~1550 tokens; at T=8192 the linear model is 2.37× faster (108.6 vs 257.7 ms). O(T) vs O(T²) showing up exactly where theory says it should. - Training speed: the big gap. 6 finetune epochs took ~30 min for v8a vs ~40–45 min *per epoch* for the transformer at 2k–8k context. That's a ~6–7× gap that gets worse with scale.
Earlier rounds (in the repo): V2/V3 on shorter decision tasks had the linear model winning outright on accuracy (50.4 vs 46.2, 66.2 vs 61.0). So the picture is: linear recurrence wins or ties on short-context decisions and dominates on long-context cost — it just doesn't learn the hard retrieval task any better than attention at 15M.
Currently running: V5, a bidirectional variant of both (15M, same budget). Will update the repo when it lands.
Happy to answer questions on the architecture or the training setup. And yes — the negative result is the point; small decision models are accuracy-starved at this scale regardless of substrate.
submitted by /u/AlarmingTrouble5261
[link] [留言]
来源：r/MachineLearning · reddit.com