Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathway
The paper argues that large-step gradient descent can rebalance multi-path deep linear networks after early symmetry breaking, unlike gradient flow predictions.
Intelligence analysis by GPT-5.4 Mini
The authors contrast continuous gradient flow with discrete gradient descent and show that large step sizes change the training dynamics. Early training still favors winner-takes-all specialization, but edge-of-stability oscillations can push the model back toward shared use of pathways.
A big team project can start with one person doing most of the work, but if the group moves in bigger jumps, the load can spread out again. The paper says that in some neural networks, a large learning step can pull work back across several paths instead of letting one path do everything.
Analysis
What the paper claims
The abstract focuses on deep linear networks with multiple pathways, where prior gradient-flow analyses predicted a winner-takes-all outcome: one pathway dominates a feature while others lose out. The paper argues that this picture changes when training uses discrete gradient descent with a large step size.
Main result
According to the abstract, single-path solutions are sharp minima, while spreading signals across multiple pathways makes the loss landscape less sharp. That reduction in sharpness becomes stronger as the number of pathways grows and as the network gets deeper. In other words, the network has a structural incentive to avoid putting everything into one path.
Training behavior
The abstract describes two phases. Early in training, the model still shows the depth-driven symmetry breaking that gradient-flow analyses predict. But later, oscillations near the edge of stability override that tendency and trigger a re-balancing phase, where signals spread back across pathways.
Why this is useful
The paper’s takeaway is that discrete optimization can behave differently from the continuous approximations often used to reason about training. Large-step gradient descent does not simply amplify specialization; in this setting, it can restore symmetry and favor shared representations instead of lasting single-path dominance. The result helps explain how depth and step size interact in pathway competition.
Key points
- The paper revisits multi-path deep linear networks and contrasts gradient flow with discrete gradient descent.
- Prior gradient-flow analyses predicted winner-takes-all specialization and broken symmetry across pathways.
- The authors argue that large-step gradient descent changes the story by reducing sharpness for distributed signals.
- Early training still shows symmetry breaking, but later oscillations can drive a re-balancing phase.
- The paper concludes that depth and step size jointly shape pathway competition and shared representations.
If the result holds more broadly, it gives researchers a clearer way to control whether networks specialize or share information. That could help explain and design training setups that keep representations more balanced in deep models.
The effect may depend on large step sizes and the specific deep linear setting studied here, so it may not transfer cleanly to more complex nonlinear models. The edge-of-stability dynamics also suggest training can become oscillatory, which may be harder to predict or control in practice.
