Orthogonal Gradient Descent (OGD)

Updated 21 February 2026

Orthogonal Gradient Descent (OGD) is a projection-based optimization technique that projects current gradients onto the space orthogonal to previously important directions.
It addresses catastrophic forgetting in continual learning by preserving prior task outputs and enhances training speed and calibration across varied neural network applications.
Variants such as Euclidean, Fisher, and component-wise OGD tailor the projection geometry to balance efficiency, robustness, and constraint satisfaction in diverse problem domains.

Orthogonal Gradient Descent (OGD) is a family of projection-based optimization algorithms designed to address catastrophic forgetting in continual learning, accelerate neural network training, improve calibration, and enable robust satisfaction of multiple constraints. OGD methods project gradient updates onto subspaces that are orthogonal to directions critical for previous tasks or model characteristics, thus ensuring selective plasticity and minimizing interference. Variants of OGD adapt the projection geometry for different problem domains, including Euclidean, Fisher (information-geometric), and component-wise manifold constraints.

1. Mathematical Foundations of Orthogonal Gradient Descent

Let $w\in\mathbb{R}^p$ denote network parameters. In sequential (continual) learning, when presented with a new task $T_N$ and loss $L_N(w)$ , standard SGD applies the unconstrained gradient $g_{\mathrm{new}} = \nabla L_N(w)$ . OGD introduces a subspace $S = \mathrm{span}\{v_1,\ldots,v_m\}$ , where each $v_i$ is a model gradient (often $\nabla f_k(x; w_{*})$ for a previous task or component). The gradient is projected onto the orthogonal complement of $S$ , yielding the update:

$P_\perp = I - A (A^T A)^{-1} A^T$

$\tilde{g} = P_\perp\, g_{\mathrm{new}} = g_{\mathrm{new}} - A (A^T A)^{-1} (A^T g_{\mathrm{new}})$

where $A$ stacks the basis vectors $v_i$ . This projection ensures that updates do not alter the outputs important for previously learned data, implementing local interference avoidance (Farajtabar et al., 2019).

OGD has variants depending on the geometry:

Euclidean OGD employs the standard inner product.
Fisher-OGD and Natural OGD utilize the Fisher information matrix as a Riemannian metric (Garg et al., 19 Jan 2026, Yadav et al., 24 Aug 2025).
Component-wise OGD orthogonalizes per-layer filter or head gradients for diversification (Tuddenham et al., 2022).

2. Algorithmic Structures and Implementation

The canonical OGD procedure comprises the following steps per task:

Gradient Computation: Calculate $g_{\mathrm{new}}$ for the current task.
Subspace Maintenance: Store model gradients from previous tasks as basis $S$ .
Projection: Project $g_{\mathrm{new}}$ onto $S^\perp$ using the efficient matrix expressions.
Parameter Update: Apply the projected gradient in place of the raw gradient.

Pseudocode (minibatch OGD):

for each minibatch:
    g = compute_gradient(loss, w)
    g_proj = g - sum((dot(g, v) / dot(v, v)) * v for v in S)
    w = w - eta * g_proj

(Farajtabar et al., 2019)

For scalable memory management, incremental PCA or approximate orthogonalization (e.g., incremental QR, SVD, or random projections) can be employed (Min et al., 2022).

When using Fisher or information-geometric projections, the projection inner product becomes $u^T F v$ , with $F$ approximated via diagonal, K-FAC, or EKFAC schemes (Garg et al., 19 Jan 2026, Yadav et al., 24 Aug 2025). Projection steps are adapted to preserve information geometry.

3. Applications: Catastrophic Forgetting, Calibration, Optimization, and Debugging

Continual Learning

OGD fundamentally addresses catastrophic forgetting by constraining updates to parameter-space directions that do not interfere with previous outputs. This method provably preserves old-task predictions (locally) and empirically achieves state-of-the-art performance on continual learning benchmarks such as Permuted MNIST, Rotated MNIST, and Split MNIST:

Method	T1	T2	T3	T4	T5	Split-MNIST (acc. %)
OGD	79.5	88.9	89.6	91.8	92.4	98.6, ..., 99.2
EWC	64.5	77.1	80.4	87.9	93.0	90.2, ..., 99.3
SGD	60.6	77.6	79.9	87.7	92.4	88.2, ..., 99.4
A-GEM	85.5	87.0	89.6	91.2	93.9	92.9, ..., 99.3

OGD closely matches or outperforms all baselines, approaching joint training upper bounds (Farajtabar et al., 2019).

Model Calibration

Layer-wise OGD, also denoted $\perp$ Grad, projects each layer's gradient off the current weights vector, optionally with norm renormalization. On semi-supervised CIFAR-10, this yields consistently improved Expected Calibration Error (ECE), lower cross-entropy loss, and higher predictive entropy, without sacrificing accuracy. Theoretical analysis shows that (non-renormalized) OGD converges to points where further loss reduction requires confidence scaling (via weight norms), rather than decision boundary modification (Hedges, 4 Jun 2025).

Optimization Speedup and Representation Diversification

OGD can be applied per-component within a layer (e.g., per convolutional filter), enforcing diversity among component gradients. This variant (OSGD) accelerates convergence—reducing epochs required by up to 50-100× compared to SGD—by ensuring intermediate features span a richer subspace. OSGD is compatible with modern optimizers (SGD, Adam, LARS) and demonstrates improved generalization and robustness to hyperparameters (Tuddenham et al., 2022).

Adversarial Constraint Satisfaction

Orthogonal Projected Gradient Descent (OPGD) enables efficient optimization under multiple constraints (e.g., attack success and detector evasion). By iteratively alternating and orthogonalizing steps w.r.t. constraint gradients, OPGD yields strictly superior escape from adversarial detectors compared to joint-loss PGD or alternating PGD. Empirically, OPGD reduces defender's accuracy to 0% at fixed detection thresholds across several state-of-the-art defenses (Bryniarski et al., 2021).

Debugging and Targeted Unlearning

OGD has been applied to neural network “debugging,” treating unlearning and relearning as a two-task continual learning scenario. By projecting debugging updates off the clean-data subspace, OGD allows targeted erasure of faulty behaviors (such as mislabeled class swaps) while maintaining generalization elsewhere, or restoration of correct decision boundaries. Experiments confirm that OGD can both entirely unlearn and relearn specific behaviors without full retraining (Chilkuri et al., 2022).

One-pass, Minimum-interference Learning

OGD provides the minimum-norm interpolant for streaming/online learning of overparameterized models, matching the end state of standard multi-pass SGD. In the “Orthogonal Recursive Fitting” (ORFit) variant, integrating OGD with incremental PCA yields an efficient, forget-free, one-pass update rule for linear and kernel models (Min et al., 2022).

4. Theoretical Guarantees and Extensions

Forgetting and Generalization

OGD is provably robust to catastrophic forgetting in the Neural Tangent Kernel (NTK) regime with infinite memory: OGD guarantees that for any previous task $k$ , the output remains unchanged after learning subsequent tasks. Generalization bounds relate cumulative transfer and task similarity in kernel space; OGD eliminates the “forgetting” term present in SGD bounds. Finite-width violations (NTK drift) reduce persistence, but can be addressed by refreshing stored gradients (Bennani et al., 2020).

Information-geometric Generalizations

Several variants augment OGD by replacing Euclidean inner products with Fisher information metrics, yielding Fisher-Orthogonal Projected Natural Gradient (FOPNG) or Orthogonal Natural Gradient Descent (ONG). These approaches unite natural gradient preconditioning with orthogonal projection, ensuring that updates are invariant under reparameterization and minimize interference in KL-divergence space. FOPNG outperforms OGD and Fisher-only NGD on realistic continual learning problems, especially when tasks are related (Garg et al., 19 Jan 2026, Yadav et al., 24 Aug 2025).

Benchmark	OGD	FNG	FOPNG
Permuted-MNIST	85.7%	84.0%	87.0%
Split-MNIST	92.0%	89.5%	96.1%
Rotated-MNIST	94.8%	90.1%	98.4%
Split-CIFAR10	81.0%	76.2%	85.7%

(Garg et al., 19 Jan 2026)

Acceleration and Orthogonality Constraints on Manifolds

Accelerated OGD on the Stiefel manifold (i.e., with explicit orthogonality constraints $X^T X = I_p$ ) generalizes Nesterov acceleration to Riemannian geometry, achieving iteration complexity $O(\sqrt{\kappa}\log(1/\epsilon))$ compared to $O(\kappa\log(1/\epsilon))$ for standard gradient descent. Accelerated OGD outperforms quasi-Newton methods on ill-conditioned matrix optimization problems (Siegel, 2019).

5. Computational Complexity and Practical Issues

OGD incurs an overhead linear in the number of stored basis vectors $M$ , with each gradient projection costing $O(Mp)$ operations. Memory requirements scale as $O(Mp)$ . In practice $M\ll p$ (e.g., $M\sim200$ ), and subsampling, pruning, approximate PCA, or blockwise strategies can further mitigate costs (Farajtabar et al., 2019, Min et al., 2022). Fisher-Ogd and ONG require estimation and inversion (or diagonalization) of Fisher matrices but apply low-rank approximations to remain computationally viable (Yadav et al., 24 Aug 2025, Garg et al., 19 Jan 2026). SVD- or QR-based orthonormalization is recommended for stability.

Component-wise OGD (OSGD) adds SVD computation per layer, with 10-20% walltime overhead on moderate CNNs, but reduces epochs needed by 20–50× (Tuddenham et al., 2022).

6. Limitations and Directions for Further Research

OGD and its variants rely on the assumption that stored basis gradients accurately represent interference directions; violations arise when the parameter space or NTK drifts, or when memory is insufficient. Adaptive basis reduction and memory-efficient sketches are ongoing research topics. Fisher-projected OGD methods offer improved theoretical properties but require hyperparameter tuning for Fisher damping and more intensive computation. Combining OGD with curvature-aware methods, structured sparsification, and methods for smooth task boundaries are active directions.

Open questions also include convergence guarantees in non-convex settings, compression of gradient bases, large-scale vision (e.g., Split CIFAR), and robust adaptation to shifting data distributions. Empirical limitations of OGD under severe over-parameterization or with small mini-batch sizes have been observed (Tuddenham et al., 2022, Bennani et al., 2020).

OGD projects intersect with adaptive filtering (recursive least-squares), where the one-pass ORFit is equivalent to zero-regularization RLS. In optimization, OGD generalizes to manifold-constrained and geometric optimization (e.g., Stiefel, Grassmannian), with Nesterov-like acceleration and Riemannian retractions (Siegel, 2019).

In adversarial robustness, OGD variants dominate joint-loss and standard PGD in multi-constraint scenarios. OGD is further applied to model debugging, targeted unlearning, and neural calibration, where it acts as a plug-and-play gradient post-processor compatible with SGD, Adam, LARS, and other optimizers (Bryniarski et al., 2021, Hedges, 4 Jun 2025).

In summary, Orthogonal Gradient Descent constitutes a broad paradigm for projection-based optimization in neural networks and beyond, facilitating continual learning without catastrophic forgetting, rapid and robust training, calibrated uncertainty estimation, and reliable constraint satisfaction through principled gradient orthogonalization. It serves as a unifying framework with strong theoretical guarantees, practical efficiency, and applicability across a wide spectrum of machine learning challenges (Farajtabar et al., 2019, Bennani et al., 2020, Bryniarski et al., 2021, Hedges, 4 Jun 2025, Tuddenham et al., 2022, Garg et al., 19 Jan 2026, Yadav et al., 24 Aug 2025, Min et al., 2022, Chilkuri et al., 2022, Siegel, 2019).