Ortho-GConv: Stabilizing Graph Neural Networks

• Ortho-GConv is an orthogonal feature transformation for GNNs that stabilizes forward signals and backward gradients, crucial for training stability.
• It employs a three-step process—hybrid weight initialization, spectral normalization with Newton iterations, and orthogonal regularization—to enforce matrix orthogonality.
• Empirical evaluations demonstrate improved convergence and accuracy across node- and graph-level tasks, mitigating early instabilities in standard GNNs.

Ortho-GConv is an orthogonal feature transformation for graph neural networks (GNNs) introduced to address instabilities in both forward normalization and backward gradients that impair training efficiency and accuracy of GNNs, especially in shallow architectures. While most prior work attributes degradation in deep GNNs to over-smoothing of node embeddings, Ortho-GConv identifies that improper linear feature transformation in standard GNN convolutional layers is the principal cause of early instability, distinct from over-smoothing effects. By enforcing orthogonality on feature transformation matrices at each layer, Ortho-GConv stabilizes magnitudes of node embeddings and preserves gradient flow, yielding improved convergence and generalization across node- and graph-level classification tasks (Guo et al., 2021).

1. Motivation and Rationale

Standard GNNs such as the Graph Convolutional Network (GCN) use a layer-wise operation of the form $$H^{(\ell)} = \sigma(\hat{A} H^{(\ell-1)} W^{(\ell)})$$, where $W^{(\ell)}$ is a learnable weight matrix. In practice, these $W^{(\ell)}$ can amplify the norm of forward signals, causing exponential growth or decay of embedding magnitudes as the layer count increases—even at moderate depths (e.g., 8 layers). Similarly, the backward gradients $\|\partial \mathcal{L}/\partial W^{(\ell)}\|_F$ fall off sharply toward lower layers, hampering effective training.

Two metrics make these phenomena quantitative:

• Forward signal magnification: $$M_{\text{sig}} = \frac{1}{|V|}\sum_i \|h^{(L)}_i\|_2 / \|h^{(0)}_i\|_2$$, ideally close to 1.
• Gradient-norm steadiness: Consistency of $\|\partial \mathcal{L}/\partial W^{(\ell)}\|_F$ across all layers.

These issues arise long before over-smoothing (convergence of node embeddings to near-constant values) becomes significant. Ortho-GConv leverages orthogonal transformations, which in convolutional and recurrent neural networks are known to preserve activation norms and gradient flow, to ameliorate both forward and backward instability in GNNs (Guo et al., 2021).

2. Mathematical Construction

Ortho-GConv maintains orthogonality of per-layer transformations via a three-part methodology:

2.1 Hybrid Weight Initialization:

For each layer $\ell$,

• Sample $P^{(\ell)}$ from standard random initialization (e.g., Glorot).
• Form $Q^{(\ell)} = \beta P^{(\ell)} + (1-\beta) I_{nn}$ for $\beta \in [0,1]$, interpolating between random and perfect orthogonality.

2.2 Orthogonal Projection:

• Spectral normalization: $W^{(\ell)}$0
• Newton iteration: Given $W^{(\ell)}$1, define
  • $W^{(\ell)}$2
  • $W^{(\ell)}$3 for $W^{(\ell)}$4
• Orthogonal weight: $W^{(\ell)}$5

2.3 Orthogonal Regularization:

A soft auxiliary loss penalizes deviation from (scaled) orthogonality for each layer:

$W^{(\ell)}$6

where $W^{(\ell)}$7 is a learnable scaling parameter (initialized to 1), and $W^{(\ell)}$8 is a small hyperparameter.

The total objective is the sum of the standard task loss ($W^{(\ell)}$9) and $W^{(\ell)}$0.

3. Algorithmic Implementation

The Ortho-GConv procedure can be summarized as follows:

1. For each layer $W^{(\ell)}$1:
   • Compute hybrid-initialized $W^{(\ell)}$2
   • Spectrally normalize and project to orthogonality via $W^{(\ell)}$3 Newton iterations
   • Set $W^{(\ell)}$4 to the resulting orthogonal matrix
2. Forward propagate via $W^{(\ell)}$5
3. Compute total loss $W^{(\ell)}$6
4. Backpropagate and update $W^{(\ell)}$7, $W^{(\ell)}$8, and all other GNN parameters with Adam optimizer

Orthogonality is enforced at every forward pass via projection, and no specialized learning rates or optimizers are required.

4. Theoretical Properties

Two theorems formalize the stability enhancements of Ortho-GConv:

• Theorem 1 (Gradient Structure): In a simplified linear GNN (no nonlinearity), the gradient with respect to $W^{(\ell)}$9 contains products of powers of $\|\partial \mathcal{L}/\partial W^{(\ell)}\|_F$0 and $\|\partial \mathcal{L}/\partial W^{(\ell)}\|_F$1, explaining why repeated application can induce vanishing or exploding gradients.
• Theorem 2 (Norm Preservation): For orthogonal $\|\partial \mathcal{L}/\partial W^{(\ell)}\|_F$2,
  1. If the input is whitened random ($\|\partial \mathcal{L}/\partial W^{(\ell)}\|_F$3), output $\|\partial \mathcal{L}/\partial W^{(\ell)}\|_F$4 remains identically distributed.
  2. The Frobenius norm of activations is preserved after transformation.
  3. The backpropagated gradient norm across $\|\partial \mathcal{L}/\partial W^{(\ell)}\|_F$5 is invariant.

These results assert that orthogonal transformations both stabilize the propagation of activation magnitudes and guard against attenuation or explosion of gradients.

5. Empirical Evaluation

Ortho-GConv was evaluated on multiple benchmark datasets:

• Node classification: Cora, Citeseer, PubMed, Cornell, Texas, Wisconsin, ogbn-arxiv

• Graph classification: D&D, PROTEINS

The method was integrated with standard GNN backbones—GCN, JKNet, GCNII for node tasks, Graph-U-Nets for graph tasks. Major findings include:

• Cora (full supervision, 2 layers): GCN baseline 85.8%, Ortho-GCN 87.3% (+1.5%).
• Cora (8 layers): GCN accuracy drops to ~81%; Ortho-GCN retains 85.3%.
• GCNII+Ortho-GConv: ∼2% gain over vanilla GCNII on average, up to 0.3% improvement on ogbn-arxiv.
• Graph classification: For D&D, Graph-U-Nets 83.0% vs. Ortho-g-U-Nets 83.9%; for PROTEINS, 77.7% vs. 78.8%.
• Stability metrics: $\|\partial \mathcal{L}/\partial W^{(\ell)}\|_F$6 remains close to 1, and per-layer gradient norms are consistent even at depth $\|\partial \mathcal{L}/\partial W^{(\ell)}\|_F$7.
• Ablation: Each component (hybrid init, orthogonal transform, regularization) is necessary; removal reduces accuracy by 1–2% on Cora.
• Newton iterations: $\|\partial \mathcal{L}/\partial W^{(\ell)}\|_F$8 is optimal for balancing accuracy and computational costs.

Dataset	Vanilla Backbone	Ortho-GConv Augmented	Absolute Gain
Cora (2 layers)	GCN: 85.8%	Ortho-GCN: 87.3%	+1.5%
Cora (8 layers)	GCN: ~81%	Ortho-GCN: 85.3%	+4.3%
D&D	Graph-U-Nets: 83.0%	Ortho-g-U-Nets: 83.9%	+0.9%
PROTEINS	Graph-U-Nets: 77.7%	Ortho-g-U-Nets: 78.8%	+1.1%

6. Practical Usage and Recommendations

Ortho-GConv is designed as a drop-in module: every linear feature transformation in any GNN architecture can be replaced by the Ortho-GConv procedure (hybrid initialization, projection, and optional regularization). Key hyperparameters and practical settings are as follows:

• Initialization blend $\|\partial \mathcal{L}/\partial W^{(\ell)}\|_F$9: ~0.4
• Newton iteration steps $$M_{\text{sig}} = \frac{1}{|V|}\sum_i \|h^{(L)}_i\|_2 / \|h^{(0)}_i\|_2$$0: 4
• Orthogonal loss weight $$M_{\text{sig}} = \frac{1}{|V|}\sum_i \|h^{(L)}_i\|_2 / \|h^{(0)}_i\|_2$$1: $$M_{\text{sig}} = \frac{1}{|V|}\sum_i \|h^{(L)}_i\|_2 / \|h^{(0)}_i\|_2$$2 or $$M_{\text{sig}} = \frac{1}{|V|}\sum_i \|h^{(L)}_i\|_2 / \|h^{(0)}_i\|_2$$3
• Optimizer: Adam with standard learning rates (e.g., 0.005 for GCNII)
• No special handling of batching or adjacency normalization required.

Integration involves:

• Replacing each $$M_{\text{sig}} = \frac{1}{|V|}\sum_i \|h^{(L)}_i\|_2 / \|h^{(0)}_i\|_2$$4 application in a GNN by Ortho-GConv's projected orthogonal layer.
• Adding the auxiliary loss $$M_{\text{sig}} = \frac{1}{|V|}\sum_i \|h^{(L)}_i\|_2 / \|h^{(0)}_i\|_2$$5 to the training objective.

The method achieves immediately more stable forward propagation and gradient signals, supports deeper or more robust shallow GNNs, and improves accuracy across diverse datasets and model backbones (Guo et al., 2021).