Pseudospectral Bounds for Transient Amplification in Coupled Gradient Descent
The paper bounds short-term growth in coupled gradient descent, showing non-normal systems can amplify before converging.
Intelligence analysis by GPT-5.4 Mini
This paper studies coupled gradient descent systems where one parameter block influences another. It shows that even when asymptotic stability holds, transient amplification can become very large, and derives pseudospectral bounds that quantify that risk.
The paper is about training loops where one part of the system nudges another part. Even if the system is safe in the long run, it can still lurch upward first, like a car that jerks forward before slowing down.
Analysis
What the paper addresses
Coupled gradient descent appears in bilevel optimization, two-time-scale stochastic approximation, and adversarial training. In these systems, the update for one block depends on another, and the Jacobian often has a block-triangular form.
The paper’s core point is that asymptotic stability is not the whole story. Even when the diagonal blocks are individually stable, the full system can show large transient amplification because the Jacobian is non-normal. That means the dynamics can grow a lot before eventually decaying.
Main result
The paper develops a pseudospectral theory for block-triangular Jacobians. Under the assumption that the diagonal blocks are symmetric with spectral radii at most γ < 1, it proves an upper bound on the Kreiss constant of the form K(J) <= 2/(1-γ) + ||C||/(4(1-γ)), where C is the coupling term. The paper also says this bound is sharp by establishing matching minimax lower bounds.
It further characterizes a critical coupling threshold for spectral instability and extends the analysis to nearly self-referential systems using a Neumann-series perturbation framework. From this, it derives a finite-horizon iteration-complexity bound of O(K(J)^2 log(1/δ)) for stochastic coupled descent.
Evidence and framing
The abstract says the theory is supported by experiments on linear-quadratic problems, IQC-based comparisons, and neural-network training. The paper frames the results as scaling laws for non-stationary two-time-scale optimization and argues that this regime is invisible to simple spectral-radius analysis.
Key points
- The paper studies transient amplification in coupled gradient descent, not just eventual convergence.
- It argues that non-normality can cause large short-term growth even when diagonal blocks are stable.
- It gives a Kreiss constant bound in terms of the coupling strength and stability margin.
- The paper extends the analysis to nearly self-referential systems with a perturbation method.
- Experiments on linear-quadratic problems and neural-network training are said to support the theory.
If the bounds hold up in practice, researchers get a clearer way to predict and control unstable-looking spikes during training. That could improve the design of bilevel and two-time-scale optimization methods by making coupling effects more measurable.
The analysis may still be limited to the kinds of structured systems studied in the paper, especially block-triangular settings with symmetric diagonal blocks. If real training setups differ too much, the bounds may be less predictive than they appear in the theory.
