Optimal Learning Rate Schedules

Updated 6 February 2026

Optimal Learning Rate Schedules are algorithmic policies that define the sequence of learning rates to enhance convergence and final generalization during iterative optimization.
They encompass classical linear and step decay methods as well as adaptive, geometric, and schedule-free approaches tailored to specific loss landscapes and problem structures.
Recent advances integrate theoretical minimax principles with data-driven automation and control-theoretic strategies to optimize performance under diverse training regimes.

Optimal learning rate schedules are algorithmic prescriptions or policies that determine the sequence of learning rates, $\{\eta_t\}$ , to be applied during iterative optimization, typically with stochastic gradient descent or its variants. The goal is to maximize convergence speed, improve final generalization, and adapt the training dynamics to the loss landscape, data, or resource constraints. Optimality can refer to theoretical minimax rates, empirical efficiency, regret minimization under non-stationarity, or adherence to control-theoretic trade-offs between effort and performance.

1. Classical Theory and Linear Decay Schedules

Classical minimax theory for convex optimization identifies linear decay as an optimal schedule under worst-case assumptions. For stochastic gradient descent with $T$ total iterations, the learning rate

$\eta_t = \eta_0 \left(1 - t/T\right) \qquad \text{with}~ \eta_0 = D / (G \sqrt{T})$

achieves the $\mathcal{O}(DG / \sqrt{T})$ optimal rate for the last iterate, where $D$ is the initial distance to a minimizer and $G$ bounds the gradient norm (Defazio et al., 2023). This linear schedule closes the sub-optimality gap of polynomial decays (such as $1/t$ or $1/\sqrt{t}$ ), particularly in large-scale or deep learning settings where final-iterate performance is required. In practice, the baseline $\eta_0$ is treated as a tunable hyperparameter, with grid search or rule-of-thumb defaults, while the decay shape $(1-t/T)$ is retained.

Theoretical refinements exploit observed, data-dependent gradient norms to construct adaptive schedules. By weighting each update inversely by squared norms, one can generate refined $\eta_t$ that encode dynamic warm-up and sharp annealing near convergence—automatically adapting to the empirical landscape (Defazio et al., 2023).

2. Geometric, Piecewise, and Problem-Adaptive Schedules

In least-squares and quadratic regimes, geometric (step) decay is near-optimal up to logarithmic corrections (Ge et al., 2019, Pan et al., 2021). The classic step-decay algorithm halves the learning rate every $S \sim T/\log T$ steps: $\eta_t = \eta_0 \cdot \alpha^{\lfloor t/S \rfloor}, \quad \alpha \in (0,1).$ This yields final excess loss scaling as $O(\sigma^2 d \log T / T)$ for least-squares regression—only a log factor from minimax rates, and strictly better than any polynomial decay for the last iterate.

For quadratic objectives with highly skewed Hessian spectra, optimal scheduling requires bucket-wise adaptation to spectral mass. The "Eigencurve" family assigns per-region inverse-time decays, balancing bias and variance across spectral groups. Empirically, practical approximations such as Elastic Step Decay and "cosine-power" decay closely mimic optimal shapes and yield state-of-the-art performance, particularly in low-epoch budgets or power-law spectral regimes (Pan et al., 2021). The standard cosine decay, $\eta_t = \eta_0 (1 + \cos(\pi t/T)) / 2$ , is nearly minimax optimal when the underlying spectrum is sufficiently skewed.

Recent theory establishes that a constant learning rate followed by linear cooldown ("wsd") produces generalization bounds matching convex theory and matches or outperforms cosine schedules in large-model LLM training, especially when continuing training or extending the training horizon (Schaipp et al., 31 Jan 2025, Defazio et al., 2023). WSD removes undesirable $\log T$ factors in final-iterate risk and provides closed-form tuning for learning rate and cooldown duration.

3. Non-Convex and High-Dimensional Regimes

In high-dimensional non-convex settings, optimal learning rate decay must balance rapid escape from rough, glassy regions with low-noise convergence in convex basins. Mean-field and Langevin analyses reveal that for a loss landscape with $p$ -spin glass structure, the optimal decay exponent is sub-linear, $\eta(t) = \eta_0 t^{-\beta}$ with $\beta_{opt} = 2/5$ for $p > 2$ (d'Ascoli et al., 2022). In problems with a planted signal, the schedule is two-phase: use a constant rate (exploration, $\beta=0$ ) until the system locates the convex basin, then decay as $1/t$ (convergence, $\beta=1$ ). Empirical regression tasks confirm these phase transitions: decaying the learning rate too early precludes recovery, while the two-phase schedule secures both exploration and sample-efficient convergence.

Advanced control-theoretic work for random feature models delivers an optimal schedule of the form $\eta^*_T(t) = T^{-\xi} (1-t/T)^\delta$ , with exponents determined by feature spectrum and teacher complexity. In the "easy" phase, $\xi = 1-b/a$ , $\delta = 2b-1$ (for feature decay rate $b$ and teacher decay rate $a$ ); in the "hard" phase (more "ill-conditioned"), optimality shifts to a warmup-stable-decay structure: use a large learning rate for most of training, then sharply anneal over a vanishing fraction of $T$ (Bordelon et al., 4 Feb 2026).

These findings expose fundamental limits of simple power-law or anytime schedules: they cannot achieve the exponents of problem-adaptive controls, and knowledge of task structure is required for truly minimax scheduling.

4. Automatic and Schedule-Free Methods

Recent work addresses the automation of learning rate schedules, either by online meta-optimization or by removing explicit scheduling. Bayesian optimization frameworks such as AutoLRS partition training into stages, using Gaussian process surrogates and exponential-loss predictors to directly minimize validation loss per stage. This approach adaptively generates stagewise-optimal learning rates, substantially accelerating deep net training across vision and NLP tasks, while removing all manual schedule design and most parameter tuning (Jin et al., 2021).

The "Schedule-Free" paradigm reparameterizes iterates so that no explicit decay or schedule is required, and yet the worst-case rates of convex theory are preserved or improved (Defazio et al., 2024). Schedule-Free SGD (and AdamW) leverages online-to-batch reductions to construct implicit last-iterate optimality, with an effective stepsize on the averaged output decaying as $1/t$. This method requires no knowledge of the training horizon, no auxiliary tuning, and no explicit schedule, matching or exceeding the performance of cosine and linear decays on 28 convex and deep-learning tasks.

Data-driven approaches such as LRTuner estimate the optimal local learning rate via a quadratic fit to loss as a function of $\eta$ on superbatches. By alternating "exploration" (increasing $\eta$ ) and "exploitation" (decreasing $\eta$ ), LRTuner facilitates escape from narrow minima and empirical bias towards wide, generalizing optima. Benchmarks show $20$– $30\%$ speedups over hand-tuned baselines on ImageNet, CIFAR-10, SQuAD, and IWSLT (Iyer et al., 2021).

Distinctively, latent ODE-based schedulers learn a dynamical systems model of training trajectories using a history of runs. By forecasting long-horizon metric evolution, these generative schedulers synthesize nonparametric, highly adaptive learning rate policies, yielding rapid convergence and flatter minima on image and LLMs (Sampson et al., 27 Sep 2025).

5. Robust Schedules Under Distribution Shift and Pruning

Learning rate schedules must also adapt to distributional nonstationarity and additional algorithmic workflows such as pruning. In online learning with distribution shift, optimal policies are derived using stochastic differential equations and Hamilton–Jacobi–Bellman equations, leading to closed-form control policies that increase the learning rate in proportion to real-time estimates of drift (the magnitude of distributional change) (Fahrbach et al., 2023). When the environment is static, the schedule smoothly decays $\eta$ to control variance; under shift, $\eta$ is rapidly increased to track the moving optimum.

In iterative network pruning, SILO prescribes an S-shaped growth in maximum learning rate from one cycle to the next, justified by theoretical reductions in hidden activation and gradient energies. The resulting schedules dynamically adapt to increasingly sparse networks, matching the upper-bound of extensive grid search ("Oracle") choices with two–four times reduced complexity. Empirical gains are pronounced at high sparsities across ResNet, VGG, DenseNet, and ViT architectures (Liu et al., 2022).

6. Modern Alternatives and Budget-Aware Profiles

Emergent designs address robustness to unknown budgets and variance in training horizon. The Reflected Exponential (REX) profile,

$\eta_t = \eta_0 \frac{1 - t/T}{1 - t/(2T)},$

was devised to interpolate between the high-lr retention of step decays (good for long budgets) and the early decay of linear schedules (good for short budgets). Across a comprehensive experimental suite, REX is rarely outperformed under any budget by classical schedules (step, cosine, OneCycle, linear), and is fully specified with no extra hyperparameters (Chen et al., 2021).

Hyperbolic and Exponential Hyperbolic Learning Rate Schedulers (HyperbolicLR, ExpHyperbolicLR) maintain stable learning curves across variable epoch settings by leveraging hyperbolic decay, where the early-training rate is asymptotically independent of the epoch budget parameter. This ensures high initial learning rates are maintained, avoiding curve "decoupling" across varying $T$ (Kim, 2024).

7. Optimal Schedules for Large-Batch and Nonconvex Training

For large-batch, nonconvex regimes relevant to deep learning, analysis of SFO complexity demonstrates that exponential schedules coupling batch size and learning rate

$b_m = b_0 \delta^m, \qquad \eta_m = \eta_0 \gamma^m, \qquad \delta \approx \gamma^2$

minimize the number of gradient evaluations to reach a target stationary-point accuracy (Umeda et al., 7 Aug 2025). Each stage operates at the variance-optimal batch size, giving geometric decay in the gradient norm and complexity $O(\log(1/\epsilon)\epsilon^{-4})$ .

A normative control-theoretic perspective defines optimal learning rate as a policy balancing immediate learning gains and effort cost. For quadratic costs and undiscounted returns, the optimal schedule is

$\mu^*(t) = \sqrt{\frac{P(T)-P(t)}{\beta}}$

with $P(t)$ the performance at $t$ and $\beta$ an effort penalty (Njaradi et al., 12 Jan 2026). This generalizes across task, optimizer, and architecture, and can be approximated in practice with episodic memory for $P(T)$ .

Summary Table: Representative Schedules and Settings

Schedule / Method	Theory Setting	Key Formula / Policy	Optimality/Empirics
Linear Decay	Convex/Lipschitz	$\eta_t=\eta_0 (1-t/T)$	Minimax, practical state-of-art (Defazio et al., 2023)
Step Decay	Strongly convex / least-squares	$\eta_t= \eta_0 \cdot \alpha^{\lfloor t/S\rfloor}$	Near minimax, O(log T) penalty (Ge et al., 2019)
Schedule-Free / Online-to-Batch	Arbitrary convex	No explicit schedule, implicit decay via averaging	Matches/beat state-of-art (Defazio et al., 2024)
Two-Phase (nonconvex)	High-dim. glassy with/without signal	Const $\to$ $1/t$; opt β from landscape $\gamma$	Theory-empirical match (d'Ascoli et al., 2022)
Eigencurve / Elastic Step / Cosine-power	Quadratic, skewed spectra	Piecewise/poly schedule by Hessian spectrum	Minimax under power-law (Pan et al., 2021)
WSD: Constant + Linear Cooldown	Non-smooth convex/LLMs	Constant rate, then linear cooldown	Removes $\log T$ , matches SOTA (Schaipp et al., 31 Jan 2025)
S-shaped (SILO, pruning)	Pruned networks, energy-theoretic	S-curve in max_lr per cycle	2–4% gain over Oracle (Liu et al., 2022)
Bayesian AutoLRS	Arbitrary, stagewise	GP/BO-based per-stage minimization	1.2–1.5× speedup (Jin et al., 2021)
REX profile	Budget-variance robustness	$\eta_t = \eta_0 \frac{1 - t/T}{1 - t/(2T)}$	70% SOTA wins across regimes (Chen et al., 2021)
Generative (latent ODE)	Data-driven, any	Data-driven ODE in observed metric/η space	SOTA, broader minima (Sampson et al., 27 Sep 2025)
Exponential LR/batch growth (SFO)	Large-batch SGD, nonconvex	$\eta_m = \eta_0\gamma^m, b_m=b_0\gamma^{2m}$	SFO-optimal in deep nets (Umeda et al., 7 Aug 2025)
Control-theoretic closed-loop	General, performance-effort tradeoff	$\eta^*(t) = \sqrt{[P(T)-P(t)]/\beta}$	Universal, memory-based implement (Njaradi et al., 12 Jan 2026)

Optimal learning rate schedules unify theoretical minimax policies with data-driven and problem-adaptive strategies, from linear and stepwise decays, to schedule-free, generative, and closed-loop approaches. Recent research demonstrates that the canonical linear decay is minimax in most standard regimes, but schedule-free online averaging, latent-ODE policy synthesis, or phase-adaptive power laws can offer further improvements, especially in the presence of non-convexity, distribution shift, or computational scale (Defazio et al., 2023, d'Ascoli et al., 2022, Defazio et al., 2024, Bordelon et al., 4 Feb 2026, Schaipp et al., 31 Jan 2025, Sampson et al., 27 Sep 2025, Liu et al., 2022, Njaradi et al., 12 Jan 2026).