Do Transformers Need Three Projections? Systematic Study of QKV Variants
A paper tests whether transformers really need separate Q, K, and V projections, and finds some shared-projection variants can match or beat the standard design.
Intelligence analysis by GPT-5.4 Mini
The paper systematically studies attention variants that tie together query, key, and value projections. Across synthetic, vision, and language benchmarks, the authors report that some shared-projection transformers perform comparably to standard QKV models while reducing memory use.
The paper asks whether a transformer really needs three different sets of knobs for attention, or if some can be shared like using one remote for several devices. The answer here is that sharing can save a lot of memory and still work surprisingly well.
Analysis
What the paper tests
Transformers usually use three separate linear projections in attention: query, key, and value. This paper asks whether all three are actually necessary, or whether some of them can be shared without hurting performance too much.
The authors evaluate three sharing schemes: Q-K=V (shared key-value), Q=K-V (shared query-key), and Q=K=V (a single projection). The latter two produce symmetric attention maps, so the paper also explores adding 2D positional encodings to recover asymmetry.
What they found
The experiments span synthetic tasks, vision benchmarks including MNIST, CIFAR, TinyImageNet, and anomaly detection, plus language modeling with 300M and 1.2B parameter models trained on 10B tokens. The paper says the shared variants can perform on par with, and sometimes better than, the standard QKV transformer.
For language modeling, the strongest result is Q-K=V, which cuts KV cache usage by 50% with only a 3.1% perplexity degradation. The authors also say projection sharing works well alongside head sharing methods such as GQA and MQA. In their reported setup, Q-K=V + GQA-4 reduces cache use by 87.5%, and Q-K=V + MQA by 96.9%, which the paper frames as useful for practical on-device inference.
Why the authors think it works
The paper argues that keys and values may live in similar representational spaces, and that attention often behaves like a low-rank system. By contrast, the Q=K-V variant is presented as breaking attention directionality.
Overall, the paper frames projection sharing as a form of weight tying in attention, with direct memory benefits for deployment-heavy settings.
Key points
- The paper tests whether transformers need separate query, key, and value projections.
- `Q-K=V` is the most promising variant in the reported language-model experiments.
- The authors report a 50% KV cache reduction with only a 3.1% perplexity hit for `Q-K=V`.
- Projection sharing can combine with GQA and MQA for much larger cache savings.
- The paper presents projection sharing as a weight-tying method with deployment benefits.
If these results generalize, smaller attention memory footprints could make strong transformers easier to run on phones and other edge devices. The paper also suggests the sharing trick can stack with GQA and MQA for even larger cache savings.
The main downside is that the gains come with some quality loss in language modeling, including a reported 3.1% perplexity degradation. The paper also shows not every sharing pattern works well, so the benefits appear sensitive to the exact attention design.
