2026 年 Post-Transformer 架构全景梳理：Mamba、RWKV、xLSTM、Titans 等对比
Mapping the Post-Transformer Architecture Landscape in 2026 [D]
一份对比表梳理了 2026 年 Post-Transformer 架构格局，涵盖 Mamba-2/Mamba-3、RWKV-7 "Goose"、BDH、xLSTM。
Went down the rabbit hole to map the different Post-Transformer architectures. Various directions include Dragon Hatchling (BDH), Mamba, RWKV, xLSTM, Titans, and a bunch of hybrid models coming from different AI Labs. My interest areas are architectural changes that can improve reasoning, memory, and adaptation within practical compute and energy budgets.
IMO, the next frontier might be less about scaling alone and more about getting more intelligence out of limited resources. That's what got me looking into these architectures and putting together this comparison table.
The 2 tables compare their core mechanisms, attention variants, test-time memory, and how some of these approaches overlap(not so mutually exclusive).
Post-Transformer architectures
| Architecture / Lab | Core mechanism |
|---|---|
| Mamba-2 / Mamba-3 - Gu, Dao et al. | Selective SSM; Mamba-3 adds complex-valued state + MIMO |
| RWKV-7 "Goose" - RWKV Foundation | Parallelizable RNN, constant memory, no KV cache |
| BDH (Dragon Hatchling) - Pathway | Graph of neurons, attention emerges from local interactions. GPU form = ReLU-low-rank + linear attention |
| xLSTM - NXAI | Exponential gating + matrix memory (mLSTM) |
| Titans - Google Research | Neural long-term memory module updated by "surprise" at test time |
| Qwen3-Next - Alibaba Qwen | Gated DeltaNet: attention, 3:1 |
| Kimi Linear - Moonshot AI | Kimi Delta Attention: MLA, 3:1 |
| MiniMax-01 / M1 - MiniMax | Lightning Attention: softmax, 7:1, + MoE |
| Jamba / Jamba 2 - AI21 Labs | Mamba : attention ≈ 7:1, + MoE |
| Nemotron-H - NVIDIA | Most attention layers swapped for Mamba-2 |
| Falcon-H1 - TII | Attention and Mamba-2 heads run in parallel inside each block |
| Zamba2 / Zamba2-VL - Zyphra | Mamba-2 backbone + shared-weight attention blocks every N layers |
| LFM2 / LFM2.5 - Liquid AI | 10 gated short-conv blocks + 6 GQA blocks, found by hardware-in-the-loop search |
| StripedHyena 2 (Evo 2) - Together AI / Arc Institute | Three Hyena conv operators (short/medium / long implicit) + attention |
| HRM / HRM-Text - Sapient Intelligence | Two coupled recurrent modules (slow planner, fast worker), single forward pass |
Where they overlap
| Architecture | Linear attn. | Softmax attn. | Test-time / fast weights | Brain-inspired |
|---|---|---|---|---|
| Mamba-2 / Mamba-3 | ✓ (via SSD) | - | - | - |
| RWKV-7 "Goose" | ✓ | - | ✓ (dynamic state evolution) | ? (RNN lineage) |
| BDH (Dragon Hatchling) | ✓ | - | ✓ (Hebbian synapses) | ✓ |
| xLSTM | ✓ (covariance update) | - | ✓ | ? (LSTM lineage) |
| Titans | - | ✓ (MAC / MAG variants) | ✓ | ✓ (memory framing) |
| Qwen3-Next | ✓ | ✓ (1 in 4) | ✓ (delta rule) | - |
| Kimi Linear | ✓ | ✓ (1 in 4) | ✓ (delta rule) | - |
| MiniMax-01 / M1 | ✓ | ✓ (1 in 8) | - | - |
| Jamba / Jamba 2 | ✓ (inherited) | ✓ (1 in 8) | - | - |
| Nemotron-H | ✓ (inherited) | ✓ | - | - |
| Falcon-H1 | ✓ (inherited) | ✓ | - | - |
| Zamba2 / Zamba2-VL | ✓ (inherited) | ✓ (shared blocks) | - | - |
| LFM2 / LFM2.5 | - | ✓ | - | - |
| StripedHyena 2 (Evo 2) | - | ✓ | - | - |
| HRM / HRM-Text | - | ? (inside modules) | - | ✓ |
For Ref: ? = uncertain or variant-dependent. “Inherited” = attributed to a constituent component. Ratios = linear-attention/SSM layers: softmax-attention layers.
I feel like a lot of major shifts in AI start as research breakthroughs and eventually become the focus of innovation across the industry. Lmk if I've missed any architectures worth adding or if there are any disagreements. Thoughts?
submitted by /u/Public-Feedback-6705
[link] [留言]
来源：r/MachineLearning · reddit.com