ResNet-SPD: Pyramidal & SPD Residual Architectures

• ResNet-SPD is a dual-framework uniting a convolutional model with pyramidal channel growth and separated stochastic depth, and a Riemannian approach handling SPD matrices using adaptive log-Euclidean metrics.
• The convolutional stream enhances spatial feature learning and model stability through gradual channel expansion and independent branch regularization, yielding superior benchmark performance.
• The Riemannian variant applies tangent-space projections and ALEM-based operations to process SPD-valued data efficiently, ensuring provable optimization on manifold-structured inputs.

ResNet-SPD refers to two distinct but thematically related architectures arising in deep learning, each leveraging the residual network paradigm but addressing different mathematical and geometric settings. In the standard convolutional setting, ResNet-SPD (originally “Deep Pyramidal Residual Networks with Separated Stochastic Depth” or PyramidSepDrop) enhances spatial feature learning and stability through a principled combination of gradual channel expansion and independent stochastic regularization across subspaces (Yamada et al., 2016). In the context of manifold-valued data, particularly Symmetric Positive Definite (SPD) matrices, ResNet-SPD denotes a residual architecture that operates within the geometry specified by Adaptive Log-Euclidean Metrics (ALEMs), generalizing conventional Euclidean residual design to Riemannian settings (&&&1&&&). The following exposition provides a comprehensive account of both paradigms, with precise architectural and theoretical detail.

1. Deep Pyramidal Residual Networks with Separated Stochastic Depth

The original ResNet-SPD, introduced as PyramidSepDrop, is a convolutional architecture that integrates two enhancements to the canonical residual network: 1) pyramidal channel expansion and 2) a separated stochastic-depth mechanism within each residual block.

Architectural Principles

• Pyramidal Channel Growth: The network uses a gradual, linear increase in channel dimensionality. The number of channels at block $l$ is $C_l = C_0 + \frac{\alpha}{L}l$, where $C_0$ is the initial width, $L$ the total number of blocks, and $\alpha$ a hyperparameter controlling overall width expansion.
• Split Residual Function: Each block's residual function $$F(x) \to \{F_{\text{lower}}(x), F_{\text{upper}}(x)\}$$ is partitioned into two branches: $F_{\text{lower}}$ retains the input dimensionality $C_{l-1}$, while $F_{\text{upper}}$ projects onto new channels added by the pyramidal growth $(C_l - C_{l-1})$.
• Separated Stochastic Depth: Independent Bernoulli masks $m_l^l$, $m_l^u$ (each parameterized by a block-dependent survival probability $p_l$) are sampled per branch during training. Forward computation per block is:

$$y_l = x_l + \frac{m_l^l}{p_l} F_{\text{lower}}(x_l) + \frac{m_l^u}{p_l} F_{\text{upper}}(x_l)$$

where $m_l^*$ are independent Bernoulli random variables; at test time, all masks are set deterministically to 1.

Separated Stochastic Depth Formulation

The probability of survival for block $l$ is $p_l = 1 - d_l$ with $d_l = \frac{l}{L} d_{\text{max}}$ ($d_{\text{max}}$ is the global death-rate). Each branch is regularized independently, mitigating vanishing gradient and overfitting when scaling to deep architectures.

Network Configurations and Empirical Performance

Implementation for CIFAR-10/100 uses pre-activation bottleneck blocks and 3×3 convolutions. Typical settings include $C_0 = 16$, $\alpha = 90$ (for 110 layers), and $d_{\text{max}} = 0.5$. Empirical results demonstrate consistent improvements over prior art:

Architecture	CIFAR-10 Error (%)	CIFAR-100 Error (%)
ResNet-110	6.43	25.16
ResDrop-110	5.23	24.58
DenseNet-100	3.74	19.25
PyramidNet-110	3.77	18.29
ResNeXt-29	3.58	17.31
ResNet-SPD-182	3.31	16.18

Ablation studies indicate:

• Naive integration of stochastic depth with PyramidNet provides no gain.
• Separated stochastic depth yields consistent improvements.
• Increased network depth offers further reductions in error, with robust scaling in multi-GPU scenarios.

Strengths and Limitations

Strengths:

• Enhanced regularization via subspace-specific dropouts.
• Improved gradient flow, especially in deep pyramidal architectures.
• Distributed training benefits from robust regularization and tolerance to smaller batch sizes.

Limitations:

• Implementation complexity increases due to additional Bernoulli masks per block.
• Tuning involves a three-way depth-width-drop trade-off.
• Stochastic depth amplifies gradient variance, requiring careful learning rate schedules for convergence (Yamada et al., 2016).

2. Riemannian Foundation: SPD Manifold Geometry and Metrics

Extending deep learning to data residing on the manifold of SPD matrices requires metrics and operations that respect the manifold’s intrinsic geometry.

Key Metrics

• Affine-Invariant Riemannian Metric (AIM):

$$g^{A}_S(V,W) = \operatorname{tr}(S^{-1} V S^{-1} W)$$

with exponential and logarithm maps defined via spectral decompositions [Pennec '06].

• Log-Euclidean Metric (LEM):

Established via the matrix logarithm diffeomorphism, yielding a flat geometry in the log-domain [Arsigny '05].

Adaptive Log-Euclidean Metrics (ALEMs)

ALEMs generalize LEM by introducing learnable bases on the spectrum: for $\alpha = (a_1, ..., a_n)$, the adaptive log-chart is

$$\phi = \mathrm{mlog}^\alpha(S) = U \cdot \text{diag}(\log_{a_i}(\sigma_i)) \cdot U^\top, \quad S = U \Sigma U^\top$$

and its inverse, $\phi^{-1}$, enabling closed-form geodesics, distance, and Fréchet mean operations. The resulting metric is:

$$g^\alpha_S(V,W) = \mathrm{tr}[(d\phi_S(V))^\top d\phi_S(W)]$$

ensuring positive definiteness and bi-invariance (Chen et al., 2023).

3. ResNet-SPD Architecture for SPD-Valued Data

Constructing a deep residual network over $(\mathcal{S}^n_{++}, g^\alpha)$, each block proceeds as:

1. Tangent-Space Projection: Project input $S$ onto $T_S\mathcal S^n_{++}$ via the chart $\Log^{\mathrm{ALE}_S}(S)$ (origin).
2. Euclidean Residual Mapping: Apply $R_W(T) = W_2 T W_2^\top + B_1$, where $W_2$, $B_1$ are Euclidean parameters.
3. Return to Manifold: Recover SPD matrix using the exponential map $\Exp^{\mathrm{ALE}_S}(R_W(T))$.

Pseudocode representation:

def SPDResidualBlock(S, W2, B1): L = Log_ALE(S) T = W2 @ L @ W2.T + B1 S_new = Exp_ALE(S, T) return S_new

Gradient computation leverages the chain rule throughout the Riemannian operations, with ALEMs’ parameters $\alpha$ updated by their own gradients.

4. Riemannian Batch-Normalization and Classification

ALEM-based batch normalization computes means and scales in the log-domain:

• Mean: $\overline{X} = \frac{1}{m}\sum_i \phi(S_i)$
• Normalization: Each $$\widetilde{X}_i = \frac{\phi(S_i) - \overline X}{\sqrt{\frac{1}{m} \sum_j \|\phi(S_j) - \overline X \|_F^2 + \epsilon}}$$
• Re-projection: $$\mathrm{BN}(S_i) = \phi^{-1}(\gamma \widetilde{X}_i + \beta)$$ with vector-valued affine parameters.

The classifier log-probes the final SPD feature and flattens for a fully connected softmax, computing categorical cross-entropy.

5. Optimization and Theoretical Guarantees

ResNet-SPD on the SPD manifold is trained by minimizing:

$$\min_{W_2, B_1, \alpha, W_c, b_c} \frac{1}{N} \sum_{i=1}^N L_\mathrm{CE}(f_\mathrm{ResNet-SPD}(S_i), y_i) + \lambda \mathcal R(\{W,B\})$$

Parameters $\alpha$ are unconstrained and receive standard Euclidean gradients.

Theoretical results guarantee:

• All geodesic computations via ALEMs are computable in the log-domain, making learning efficient.
• ALEM metrics form a bi-invariant Riemannian family, admitting closed-form means/distances.
• The Riemannian isometry of $\phi^\alpha$ to $(\mathcal S^n, \|\cdot\|_F)$ implies convergence properties equivalent to standard Euclidean architectures under SGD (Chen et al., 2023).

6. Significance and Applications

ResNet-SPD architectures have set new benchmarks in both conventional and Riemannian deep learning contexts. For object recognition with Euclidean data, the separated stochastic depth in pyramidal architectures improves regularization and scalability. For learning on SPD-valued data—ubiquitous in fields such as brain connectomics, diffusion tensor imaging, and covariance representations—ALEM-based residual networks permit principled geometric learning with provable properties and adaptivity.

A plausible implication is that future advances might continue unifying architectural innovations from standard residual networks (such as channel-growth and stochastic depth) with geometric deep learning for manifold-valued data, leveraging the theoretical guarantees of Riemannian metrics by design.