Tensor Cache: Eviction-conditioned Associative Memory for Transformers
A new two-level cache aims to keep transformer memory bounded without losing useful evicted tokens. The paper says it improves the tradeoff between memory use and model quality.
Intelligence analysis by GPT-5.4 Mini
Tensor Cache pairs exact sliding-window attention with a second memory that stores evicted key-value pairs as compressed outer products. The paper also says it fixes a training shortcut that adds unwanted cross-token terms and shows better memory-quality results than bounded-state baselines.
A transformer is like a reader with a backpack for notes. The bigger the backpack, the more it can remember, but it gets heavy fast.
This paper gives the reader two pockets. The front pocket keeps the newest notes exactly, and the back pocket stores old notes in a smaller, compressed way instead of throwing them away.
The authors say this helps the model remember useful older information without using too much memory. They also fix a shortcut that can mix notes together in the wrong way.
Analysis
What the paper proposes
Autoregressive transformers normally keep a KV cache that grows with context length. The paper says sliding-window attention controls memory, but it also throws away anything outside the window, even if that older information still matters. Tensor Cache is proposed as a two-level design: a first-level cache (L1) for recent tokens using exact sliding-window softmax attention, and a second-level cache (L2) for evicted tokens.
How L2 works
The outer-product memory stores evicted key-value pairs in a fixed-size per-layer matrix. Future queries can read from that matrix with a single matrix multiplication, using the linear-attention identity the abstract gives as (q_t(k_i \otimes v_i)=\langle q_t,k_i\rangle v_i). A learned scalar gate combines the L1 and L2 outputs, and the paper says per-head decay and write-rate parameters are trained end to end.
Training detail the paper calls out
The abstract highlights a problem with a common chunked-mean shortcut. It says updating memory with chunk averages can silently introduce many spurious cross-token outer products within a chunk. Tensor Cache instead uses a parallel weighted-sum scan that the paper describes as equivalent to per-token writes within float32 epsilon.
Claimed results
Across systems scaling, associative recall, long-context language modeling, and memory-capacity tests, the paper says Tensor Cache improves the memory-quality frontier over bounded-state baselines. The main contribution is not the outer-product memory itself, but the use of that memory as a cache fed only by sliding-window evictions.
Key points
- Tensor Cache uses exact sliding-window attention for recent tokens and a second memory for evicted tokens.
- Evicted key-value pairs are compressed into a fixed-size outer-product matrix and read with a matrix multiply.
- A learned gate mixes the short-term and long-term memories.
- The paper says a common chunked-mean training shortcut creates spurious cross-token terms.
- The authors claim better memory-quality tradeoffs than bounded-state baselines.
