Last Iterate Convergence of AdaGrad-Norm for Convex Non-Smooth Optimization

Abstract: We study the convergence of the last iterate (i.e., the $(N+1)$-th iterate) of the AdaGrad method. Although AdaGrad -- an adaptive subgradient method -- underpins a wide class of algorithms, most existing convergence analyses focus on averaged (or best) iterates. We derive worst-case upper bounds on the suboptimality of the final point and show that, with an optimally tuned stepsize parameter, the last iterate converges at the rate $O(1/N^{1/4})$. We complement this guarantee with matching lower-bound constructions, proving that this rate is tight and that AdaGrad's last-iterate rate is strictly worse than the classical $O(1/N^{1/2})$ rate for its averaged iterate. Technically, our analysis introduces an exponent parameter that captures the growth of the cumulative squared subgradients; combined with the last-iterate inequality of Zamani and Glineur (2025), this reduces the problem to bounding a particular series.

• The paper establishes that the last-iterate suboptimality of AdaGrad-Norm decays at O(N^(-1/4)) with horizon-dependent stepsizes, revealing an inherent adaptivity penalty.
• The paper derives matching upper and lower bounds by introducing a growth exponent δ for cumulative subgradients, and validates these findings through empirical error analysis.
• The paper contrasts the slower last-iterate performance with averaged iterates and non-adaptive methods, emphasizing the need for horizon-aware adjustments in adaptive optimization.

Last Iterate Convergence of AdaGrad-Norm for Convex Non-Smooth Optimization

Introduction and Problem Formulation

This paper rigorously investigates the convergence rate of the last iterate of the AdaGrad-Norm (scalar AdaGrad) method for deterministic convex non-smooth optimization. While extensive literature addresses the averaged or best iterate convergence of AdaGrad-type methods, the last-iterate behavior—which aligns with typical practical usage—has remained insufficiently characterized, especially in adaptive scenarios without explicit horizon-dependent decay.

Formally, the focus is on minimizing a convex objective $f$ over a closed convex set $X$ under the assumption that all subgradients are $G$-bounded. The method under study is the projected AdaGrad-Norm algorithm with the standard adaptive stepsize:

$$h_k = \frac{h}{\sqrt{G^2 + \sum_{i=1}^k \|g^i\|^2}},$$

with $h$ typically set to depend on the total number of iterations $N$ via $h = R/N^\gamma$. The convergence guarantee for the last iterate $x^{N+1}$, as opposed to the canonical averaged iterate, is the main object of analysis.

Main Theoretical Results

Tight Upper and Lower Bounds

The principal upper bound established shows that when $h = R / N^\gamma$ for any $\gamma\in(0, 1/2]$, the worst-case last-iterate suboptimality decays as

$X$0

The minimal rate is realized at $X$1, yielding $X$2 for the minimax-optimal worst-case decay. Notably, this is asymptotically slower than the $X$3 rate for the averaged iterate of (scalar) AdaGrad (2604.10728).

The architecture of the proof introduces an exponent $X$4 that quantifies the growth of the cumulative squared subgradients: $X$5, with $X$6. By parameterizing the trajectory this way and exploiting a refined last-iterate inequality [Zamani et al.], the analysis reduces to bounding a sum of the form

$X$7

over admissible sequences $X$8, culminating in tight asymptotics for the bound.

A matching lower bound is constructed: for any stepsize exponent $X$9, instances exist such that the last-iterate suboptimality cannot be improved beyond the stated rates, cementing the $G$0 bound as tight for horizon-dependent (non-decaying) AdaGrad stepsizes.

Figure: Empirical Verification

Figure 1: Empirical last-iterate error and the theoretical bound for AdaGrad with fixed values of $G$1 and various $G$2.

Empirical results in Figure 1 demonstrate that the theoretical bounds closely track the observed error curves for carefully controlled adversarial instances. The dependence of convergence rates on both the stepsize exponent $G$3 and the subgradient growth parameter $G$4 is quantitatively confirmed.

Figure 2: Empirical last-iterate error and theoretical bound with window maximum smoothing applied for visualization robustness.

The smoothing applied in Figure 2 (via window maxima) further clarifies the scaling behavior and enables accurate estimation of the convergence exponent, supporting the theoretical prediction. Detailed experiments corroborate that, as the exponent $G$5 varies, the empirical slopes align with their theoretical counterparts.

Comparative Perspective and Theoretical Implications

The analysis highlights a key distinction: even with optimal stepsize tuning, AdaGrad's last-iterate convergence for standard scalar rules remains strictly suboptimal compared to averaged versions—a contrast to horizon-dependent non-adaptive methods, which can achieve minimax rates for the last iterate with suitable step decay (e.g., linearly decreasing stepsizes yield $G$6).

Contrasting AdaGrad-Norm (scalar) with constant stepsize projected subgradient methods, one finds that even the latter admits a slightly faster $G$7 rate for the last iterate, up to log factors. Thus, adaptivity through standard AdaGrad-Norm, while yielding practical advantages, imparts a fundamental last-iterate penalty in the convex non-smooth regime.

The paper's technical apparatus—specifically, the introduction and exploitation of the $G$8 parameter—offers a new analytical device for quantifying adaptivity-induced effects in worst-case optimization literature. This framework generalizes to more intricate adaptive algorithms, potentially supporting more fine-grained lower and upper bound analyses for variants of AdaGrad, RMSProp, and Adam.

Practical Implications and Future Directions

Practically, the results suggest that, for applications requiring strong worst-case guarantees for the final iterate (rather than averages), standard AdaGrad-Norm—without further stepsize modifications—may be disadvantageous relative to either last-iterate-tuned non-adaptive schemes or averaged AdaGrad. This has direct implications for deployment in large-scale learning systems and underscores the need for horizon-aware or modified adaptive learning rates when last-iterate performance is critical.

Theory-oriented future work may address several open directions:

• Extending the tight last-iterate analysis to the coordinate-wise (diagonal) AdaGrad setup.
• Characterizing last-iterate rates in stochastic or smooth convex settings, with or without heavy-tailed gradient noise.
• Investigating the interaction of adaptivity, horizon-dependence, and last-iterate performance for Adam, RMSProp, and related algorithms.

Conclusion

This paper delivers a comprehensive tight characterization of the last-iterate convergence behavior of AdaGrad-Norm in convex non-smooth optimization, demonstrating a sharp $G$9 minimax rate for horizon-dependent stepsizes. This result exposes a genuine adaptivity penalty inherent in the standard AdaGrad-Norm update when considering worst-case final-point guarantees, distinguishing it both from its averaged iterate counterpart and horizon-aware non-adaptive procedures. The technical approach, based on problem-dependent growth exponent analysis, offers a versatile foundation for future generalizations and analysis of related adaptive methods.