Yandex published interesting research, and it exploits the KV cache to do something super cool. Here’s the gist. It is long, but I promise it’s super interesting. So, buckle up.
Start with what the KV cache normally does.
When a model generates text, it does not want to recompute everything it already read every single time it produces a new word. So it stores the internal representation of everything so far, and just reuses it. That storage is the KV cache.
Most people, myself included, think of it purely as a speed optimization. Nothing more.
But here’s what they did with it…
The KV cache is not just stored data; it is the model’s active memory of the conversation. It decides what the model can see and in what order.
And if that is true, then changing how the cache is organized changes what the model is capable of, even if you never touch the model’s weights.
So how do you change it.
The trick (as mentioned in the paper) is that a chunk of that stored memory can be kept in one place (shared across), but shown to different readers in a different order.
Imagine two people looking at the same shared notebook, except each person can be handed the pages in a different sequence. Nothing in the notebook is copied or rewritten, only the order each reader sees it in changes. That is the whole mechanism.
Once you have that, a few things become possible that were not possible before.
-
You can run several model instances against the same shared memory, so each one sees what the others are doing as they do it, instead of waiting for a full answer before reacting.
-
One model can play two roles off the same memory: one part keeps thinking, another part talks. The talking part does not wait for the thinking part to finish; it just uses what is ready so far. And the thinking part can see what has already been said, so it can adjust as it goes.
Yandex showed a demo of an agent that played Doom. The model gets a constant stream of game frames while it is reasoning and while it is choosing actions, all happening over the same shared memory setup.
Nobody trained this model on Doom. The only thing that was built was the way its memory is organized and exposed to it, and that alone was enough to make it act in real time.
The way I would summarize it is this.
Interactivity did not come from retraining a model to be interactive. Some of it can come from changing how a model’s own memory is arranged and shared, which is a much cheaper lever to pull than most people assume.