我们是在靠“记住更多”解决 LLM 记忆问题吗？真正的问题或许是知道该忘记什么

Long-context models can now handle hundreds of thousands, even millions of tokens. But I've been wondering if we're focusing too much on recall and not enough on what happens when information changes.

Say a coding agent reads that an API timeout is 30 seconds. Later, the documentation changes it to 5 seconds. Both versions are still in context, but the agent keeps using 30.

It hasn't forgotten anything. It just hasn't updated what it knows.

I've been reading about different memory architectures, and this distinction seems pretty important.

With Transformers and RAG, old and new information can coexist, and the model has to resolve the conflict when reading it. In simple linear attention, associations accumulate in a shared state, so writing something new doesn't necessarily remove what's already there. DeltaNet tries to address this through prediction-error-based updates. BDH (Dragon Hatchling) takes another route with Hebbian-style synaptic plasticity.

But even with these approaches, there's still the question of when to update something, when to keep it, and how to avoid messing up unrelated memories.

I also wonder if we're testing this properly. Most long-context evaluations focus on whether a model can find information. What if we instead tested whether it changes its answer when given newer, contradictory evidence, while still remembering what was previously true?

Curious what people here think. Is this mostly a problem with how memory is stored and updated, or with how models reason over conflicting information?

Also, are there any good benchmarks specifically targeting this? Would be interested in reading them.

submitted by /u/Prestigious-Tip6030
[link] [留言]