Tracking SGD Performance Metrics

• Tracking performance of SGD is the process of quantifying key metrics such as loss interpolation, convergence phases, and hyperparameter dynamics.
• It leverages interpolation between iterates to extract metrics like valley floor location, step distance, and gradient angles for diagnostic insights.
• Practical strategies integrate real-time metric logging and adaptive hyperparameter adjustments to balance exploration and convergence.

Stochastic Gradient Descent (SGD) is a foundational optimization algorithm in large-scale machine learning and signal processing. Tracking the performance of SGD involves rigorous quantification of its optimization trajectory, convergence phases, error dynamics, and generalization behavior as a function of key hyperparameters—most notably learning rate, batch size, step size schedules, and data dependence structure. This article surveys formal methodologies, empirical findings, and theoretical frameworks for tracking the performance of SGD in both convex and nonconvex regimes.

1. Trajectory Tracking and Loss Surface Interpolation

A precise understanding of how SGD moves through the high-dimensional loss landscape is achieved by interpolating the loss between successive iterates and quantifying geometric properties along this path. For parameter vectors $\theta_t$ and $\theta_{t+1} = \theta_t - \eta g_t$ at iteration $t$, one constructs the one-dimensional path

$$\theta(\alpha) = (1 - \alpha) \theta_t + \alpha \theta_{t+1}, \qquad \alpha \in [0, 1].$$

Full-batch losses $L(\theta(\alpha))$ evaluated over a grid of $\alpha$ allow for computation of several key metrics:

• Valley floor location: $\theta^* = \theta(\alpha^*)$ for $$\alpha^* = \arg \min_{\alpha \in [0, 1]} L(\theta(\alpha))$$.
• Height above valley floor: $$h_t = [L(\theta_t) + L(\theta_{t+1}) - 2 L(\theta^*)]/2$$.
• Step distance: $d_t = \|\theta_{t+1} - \theta_t\|_2$.
• Gradient angle: $\theta_{t+1} = \theta_t - \eta g_t$0.
• Net displacement: $\theta_{t+1} = \theta_t - \eta g_t$1.

Empirically, loss interpolations are nearly convex, and SGD typically moves at a fixed height $\theta_{t+1} = \theta_t - \eta g_t$2 above the valley floor, "bouncing" between valley walls. This mechanism allows SGD to traverse the landscape by jumping over small barriers, enhancing exploration, especially for small batch sizes and large learning rates. Monitoring these metrics epoch-wise provides practitioners with direct diagnostics of the quality and breadth of parameter exploration, which correlates strongly with generalization performance (Xing et al., 2018).

2. Influence of Hyperparameters and Noise Structure

The batch size $\theta_{t+1} = \theta_t - \eta g_t$3 and learning rate $\theta_{t+1} = \theta_t - \eta g_t$4 modulate the stochastic dynamics of SGD according to the decomposition

$\theta_{t+1} = \theta_t - \eta g_t$5

where $\theta_{t+1} = \theta_t - \eta g_t$6 is the gradient noise covariance. The characteristic "temperature" is $\theta_{t+1} = \theta_t - \eta g_t$7.

The scaling laws and qualitative effects are as follows:

• Learning rate $\theta_{t+1} = \theta_t - \eta g_t$8: Determines the typical height $\theta_{t+1} = \theta_t - \eta g_t$9 above the valley floor. Large $t$0 maintains exploration over barriers, enabling entry into flatter regions.
• Batch size $t$1: Smaller $t$2 injects more structured gradient noise, increasing spread in parameter space and decreasing $t$3 (less back-and-forth oscillation, more stochastic exploration).
• Noise structure: Structured gradient noise (covariance $t$4) is essential for exploration along sharp directions, preventing collapse into narrow valleys. Isotropic artificial noise added to GD degrades generalization (Xing et al., 2018).

The phase diagram in the $t$5 plane separates three dynamical regimes:

Regime	Conditions	Generalization Error Scaling
I. Noise-dominated	$t$6, $t$7	$t$8
II. First-step-dominated	$t$9, $$\theta(\alpha) = (1 - \alpha) \theta_t + \alpha \theta_{t+1}, \qquad \alpha \in [0, 1].$$0	$$\theta(\alpha) = (1 - \alpha) \theta_t + \alpha \theta_{t+1}, \qquad \alpha \in [0, 1].$$1 a function of $$\theta(\alpha) = (1 - \alpha) \theta_t + \alpha \theta_{t+1}, \qquad \alpha \in [0, 1].$$2 only
III. GD-like	$$\theta(\alpha) = (1 - \alpha) \theta_t + \alpha \theta_{t+1}, \qquad \alpha \in [0, 1].$$3	$$\theta(\alpha) = (1 - \alpha) \theta_t + \alpha \theta_{t+1}, \qquad \alpha \in [0, 1].$$4 independent of $$\theta(\alpha) = (1 - \alpha) \theta_t + \alpha \theta_{t+1}, \qquad \alpha \in [0, 1].$$5

where $$\theta(\alpha) = (1 - \alpha) \theta_t + \alpha \theta_{t+1}, \qquad \alpha \in [0, 1].$$6 is the effective margin/curvature scale of the loss and $$\theta(\alpha) = (1 - \alpha) \theta_t + \alpha \theta_{t+1}, \qquad \alpha \in [0, 1].$$7 is a critical batch size determined by problem hardness $$\theta(\alpha) = (1 - \alpha) \theta_t + \alpha \theta_{t+1}, \qquad \alpha \in [0, 1].$$8 and dataset size $$\theta(\alpha) = (1 - \alpha) \theta_t + \alpha \theta_{t+1}, \qquad \alpha \in [0, 1].$$9 (Sclocchi et al., 2023).

3. Convergence Detection, Tracking, and Error Bounds

SGD's evolution passes through distinct phases:

• Transient phase: Rapid motion towards minimizers; bias dominates. For $L(\theta(\alpha))$0 steps, $L(\theta(\alpha))$1.
• Stationary phase: Small-scale oscillations bounded in a $L(\theta(\alpha))$2 ball around the minimum. Variance dominates, and progress plateaus.

An algorithmic test for entry into stationarity uses the inner product between successive stochastic gradients $L(\theta(\alpha))$3. The running sum $L(\theta(\alpha))$4 transitions from predominantly positive in the transient phase to negative in stationarity, with the sign change acting as a robust stopping/convergence criterion (Chee et al., 2020).

Under strong-convexity, the excess risk can be tightly bounded. In the streaming approximation regime with drifting targets, expected tracking error for OLS (parameter $L(\theta(\alpha))$5) is $L(\theta(\alpha))$6 in both expectation and high probability with optimal step-size, even when the least-squares solution itself drifts over time (Korda et al., 2013).

For non-convex matrix recovery, the Alecton SGD tracking algorithm tracks the evolving subspace. With an adaptive step-size, subspace error decays as $L(\theta(\alpha))$7 iterations, constant rank, and random initialization—requiring only second-moment conditions on the sampling distribution (Sa et al., 2014).

4. Step-Size Schedules and Multi-Epoch Dynamics

Step-size schedule critically impacts not only convergence rates but also the smoothness and trackability of SGD's path:

• Classical decay: $L(\theta(\alpha))$8 is standard but can result in oscillatory late-stage behavior.
• Log-augmented decay: $L(\theta(\alpha))$9 decays slightly faster, yielding convergence rate $\alpha$0 for smooth nonconvex losses. This schedule smooths late-phase oscillations and improves final test accuracy, as shown empirically on deep nets and kernel SVMs (Shamaee et al., 2023).

Multi-epoch training substantially improves excess risk convergence in problems satisfying the Polyak–Łojasiewicz (PL) condition. After the first pass ($\alpha$1 convergence), subsequent passes over the data lead to noise decay $\alpha$2, enabling convergence as fast as $\alpha$3 after $\alpha$4 passes. The acceleration ceases beyond this saturation point. The phenomenon is theoretically confirmed for least-squares losses and requires PL-like curvature near the minimizer (Xu et al., 2021).

5. Tracking in Streaming, Non-i.i.d., and Dependent Regimes

Real-world data streams often exhibit temporal dependence, bias, and distributional shift. In this context, mini-batch SGD with time-varying batch-size, combined with Polyak–Ruppert averaging, offers robust tracking:

• Time-varying mini-batch size ($\alpha$5) breaks long- and short-range dependence, with error terms exhibiting accelerated decay rates compared to constant-batch SGD.
• Polyak–Ruppert averaging achieves the Cramér–Rao lower bound $\alpha$6 for the mean-squared error in well-behaved regimes, regardless of moderate bias or dependence, ensuring robust parameter tracking even under weak mixing (Godichon-Baggioni et al., 2022).

The leading error components decompose into initial-condition, bias, and variance, each controllable via step-size $\alpha$7 and batch-size schedule parameters $\alpha$8.

6. Practical Monitoring, Diagnostics, and Implementation Strategies

Effective tracking and real-time performance prediction of SGD require systematic metric logging and tailored adjustment strategies:

• Log trajectory metrics (loss, validation accuracy, $\alpha$9, step-size) per batch or epoch.
• Visualize $\theta^* = \theta(\alpha^*)$0, $\theta^* = \theta(\alpha^*)$1, $\theta^* = \theta(\alpha^*)$2, and distance-from-initialization to diagnose stagnation or underexploration.
• Leverage the temperature $\theta^* = \theta(\alpha^*)$3 as a diagnostic of SGD's dynamical regime; maintain $\theta^* = \theta(\alpha^*)$4 near the optimal range for noise-dominated or first-step-dominated performance (Sclocchi et al., 2023).
• Use warm restarts or cyclical boosts to escape narrow valleys, and adapt batch size downward if displacement from initialization saturates too low (Xing et al., 2018).
• Employ automatic learning-rate tuning: reduce $\theta^* = \theta(\alpha^*)$5 each time stationarity is detected for robust, hand-off scheduling (Chee et al., 2020).

Rigorous metric tracking in these frameworks yields actionable insight into the transient and stationary behaviors of SGD, providing the foundation for adaptive control and performance prediction in both classical and deep-learning applications.

---

References:

• "A Walk with SGD" (Xing et al., 2018)
• "Understanding and Detecting Convergence for Stochastic Gradient Descent with Momentum" (Chee et al., 2020)
• "Why Does Multi-Epoch Training Help?" (Xu et al., 2021)
• "Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscape" (Kamali et al., 2023)
• "Modified Step Size for Enhanced Stochastic Gradient Descent: Convergence and Experiments" (Shamaee et al., 2023)
• "Fast gradient descent for drifting least squares regression, with application to bandits" (Korda et al., 2013)
• "Global Convergence of Stochastic Gradient Descent for Some Non-convex Matrix Problems" (Sa et al., 2014)
• "Learning from time-dependent streaming data with online stochastic algorithms" (Godichon-Baggioni et al., 2022)
• "On the different regimes of Stochastic Gradient Descent" (Sclocchi et al., 2023)