WAV: Multi-Resolution Block Residual Routing for Deep Decoder-Only Transformers
WAV v1 adds directional residual details to deep decoder-only Transformers and improves results most clearly at 48 layers.
Intelligence analysis by GPT-5.4 Mini
The paper argues that block-level residual sums lose useful structure, so it adds two directional bases to preserve attention-vs-MLP and early-vs-late dynamics. In deep models, that extra detail helps routing work better with almost no added parameters.
The paper says a deep language model is like a stack of many tiny workers, and one summary number for each floor can miss important details. WAV adds a couple of extra notes about what happened in each block, which helps the model make better choices in bigger stacks.
Analysis
What the paper proposes
Residual connections are essential in deep Transformers, but the authors say standard PreNorm residual streams add updates with fixed weights. They build on recent attention residual routing methods, which use content-dependent depth-wise mixing, and on block attention residuals, which compress routing into block-level summaries.
The main idea
The paper argues that a single block summary hides directional structure. WAV v1 adds two extra bases to each block: a phase basis that contrasts attention and MLP updates, and a split basis that contrasts early and late sublayer updates. These detail signals travel alongside the usual block summary through the same depth-wise softmax mixer. The method also uses negative detail-source initialization and detached RMS matching to stabilize training.
Reported results
On character-level TinyStories and Text8, the method is not consistently better at 12 layers. The benefit becomes clearer as depth increases: at 24 layers it is competitive, and at 48 layers it beats the baselines reported in the abstract. The paper says WAV v1 lowers validation loss from 0.4960 to 0.4738 on TinyStories and from 0.9363 to 0.9305 on Text8 at 48 layers, with negligible extra parameters.
Takeaway
The authors’ claim is narrow but important: deeper residual routing seems to need more than a single block sum. Preserving directional residual details may matter when scaling decoder-only Transformers.
Key points
- WAV v1 adds multi-resolution residual routing to decoder-only Transformers.
- It keeps directional detail about attention vs. MLP and early vs. late sublayer updates.
- The method is most effective in deeper models, especially at 48 layers.
- The paper reports lower validation loss than Block AttnRes on TinyStories and Text8 at that depth.
- The authors say the extra cost is negligible.
If the method holds up beyond these tests, it could offer a cheap way to make deep decoder-only Transformers train better. The reported gains at 48 layers suggest that preserving more residual structure may help scaling without adding much overhead.
The paper’s own results show the method is not consistently helpful at 12 layers, so the benefit may depend on depth. The evidence in the abstract is also limited to two character-level language modeling benchmarks, so broader usefulness is still unproven.
