Unlocking Feature Learning in Gated Delta Networks at Scale
The paper derives scaling rules for Gated Delta Networks and says they enable stable learning-rate transfer across widths in language-model pre-training.
Intelligence analysis by GPT-5.4 Mini
This arXiv paper extends the idea behind maximal update parametrization to a more complex linear architecture, Gated Delta Networks. The authors say their scaling rules make learning-rate transfer work across model widths, while standard parametrization does not.
The paper is about teaching a complicated model to keep working well as it gets bigger. It gives rules for picking training settings so the model does not need a fresh tune-up each time its size changes, like a bike that can be adjusted once and then fit many riders.
Analysis
What the paper is doing
The paper focuses on a practical scaling problem in large language model training: how to choose hyperparameters so that a model trained at one width still behaves well at another width. The authors point out that maximal update parametrization (\muP) has already helped with zero-shot hyperparameter transfer in standard Transformers, but that extending similar ideas to linear models with structured state transitions and more complicated architectures has been less explored.
Main claim
For Gated Delta Networks, the authors say they derive scaling rules by carefully tracking coordinate-size estimates through the forward pass, the gating mechanism, and the recurrent state dynamics. The abstract says these rules are not just theoretical; experiments on language-model pre-training show stable learning-rate transfer across model widths under both AdamW and SGD.
Why that matters
The paper contrasts its configurations with standard parametrization, which the abstract says fails to transfer. That makes the result more than a narrow optimization tweak: it is presented as evidence that the analysis correctly captures the network’s scaling behavior and can improve practical training setup.
The contribution is therefore twofold: it gives a principled scaling analysis for a less conventional architecture, and it reports that the resulting setup works across different optimizers and model widths in pre-training experiments. The abstract does not claim broader language-model quality gains, so the strongest supported takeaway is about stable transfer and training reliability rather than a universal improvement in performance.
Key points
- The paper studies how to scale Gated Delta Networks without retuning everything from scratch.
- It extends ideas from maximal update parametrization to a more complex architecture with gating and recurrent state dynamics.
- The authors say their derived scaling rules support stable learning-rate transfer across widths.
- Experiments on language-model pre-training reportedly work with both AdamW and SGD.
- Standard parametrization is said to fail to transfer across widths in these experiments.
If the analysis generalizes, teams could train Gated Delta Networks with less trial-and-error when changing width. The paper also suggests the same setup can transfer learning rates under both AdamW and SGD, which could make scaling more predictable.
The abstract only supports these results for Gated Delta Networks in language-model pre-training, so the rules may not carry over cleanly to other architectures or tasks. It also shows that standard parametrization still fails to transfer, which means careless setup can still break scaling.
