Residual-Free Architecture

• Residual-Free Architecture is a design paradigm that eliminates explicit skip connections by using analytical or structural reparameterizations to maintain stability and efficiency.
• Methodologies like RMNet block conversion and bubble-enriched FEM employ spectral and variational corrections to ensure robust performance even under extreme conditions.
• Empirical evaluations demonstrate that these architectures achieve competitive accuracy and enhanced pruning capabilities compared to traditional residual-based models.

A residual-free architecture is any computational or learning system that purposefully eliminates explicit residual (skip) connections or replaces them with equivalent transformations, while maintaining stability, accuracy, and computational efficiency. This design principle appears in distinct fields, including numerical methods for PDEs, convolutional neural networks, and transformer-based models. The motivation may include stabilization, pruning amenability, architectural simplification, or restoration of feature hierarchy. Research demonstrates that, given the appropriate adjustments, many advantages of residual architectures can be replicated or exceeded without explicit skip connections.

1. Theoretical Foundations

In neural network and PDE communities, “residual” connections create pathways that bypass one or more computational layers, facilitating gradient flow or stabilization. Residual-free architectures, by contrast, either entirely remove or analytically absorb these pathways. For instance, in deep learning, the block output transforms from $y = x + F(x)$ to a pure feed-forward sequence. In high-Péclet finite element contexts, the stabilization provided by residual terms is replaced via functional enrichment of basis spaces.

In transformers, the absence of residuals fundamentally alters gradient propagation. The Jacobian of the parameter-to-output mapping becomes the product of block derivatives without spectral regularization from additive identity terms. Direct composition of ill-conditioned self-attention and MLP operators without spectral shifting leads to rapidly deteriorating conditioning unless addressed by principled initialization strategies (Ji et al., 30 Sep 2025).

2. Methodologies for Residual-Free Systems

2.1 Spectral and Variational Correction in PDEs

Advection-dominated advection-diffusion problems are prone to spurious oscillations in standard low-order Galerkin FEM. The residual-free bubble methodology (Kryven et al., 2016) circumvents this by enriching the finite element space with elementwise bubble functions, satisfying homogeneous Dirichlet conditions on element boundaries. Formally, for a partition $\mathcal{T}_h$ of $\Omega$, the trial space is $V^h = V_1^h \oplus B^h$, with $B^h$ consisting of compactly supported, zero-boundary "bubble" functions. Local sub-element correction is computed by solving on each $K$:

$$- k \Delta u^{b}_K + w \cdot \nabla u^{b}_K = f - (-k \Delta u^1_K + w \cdot \nabla u^1_K)\,,\;\;u^b_K|_{\partial K}=0.$$

A spectral basis, such as functions $M_{i,j}^r$ on the reference element, enables polynomial-exact quadrature and diagonal dominance of local stiffness matrices.

2.2 Analytical Elimination in Convolutional Networks

The RMNet framework (Meng et al., 2021) provides a blockwise algorithm for transforming any standard ResNet into a pure feedforward architecture while preserving its output. It replaces the parallel branch-and-add topology via a “reserve and merge” operation:

• The "reserve" step explicitly concatenates the original branch with an identity-mapped channel expansion, guaranteeing that $x$ is fully preserved along the computational path.
• The "merge" step involves a modified convolution fusing intermediate and reserved signals before the final activation, so that $\tilde{y} = M(A(x)) \equiv x + F(x)$. This transformation is structurally exact for standard ResBlocks and produces functionally identical outputs without runtime addition.

2.3 Conditioning and Initialization for Skipless Transformers

Residual-free transformer architectures encounter problematic gradient conditioning due to the accumulation of singular values in repeated blockwise Jacobian products. To counteract this:

• Self-attention value/output projection matrices are initialized scale-orthonormal ($W^V_\ell = c U,\, W^O_\ell = c V^\top$ with $U,V$ from SVD), ensuring the dominant derivative term is nearly unitary.
• Query/key projections are generated to be diagonal dominant ($W^Q W^{K\top} \approx \beta I + \alpha Z$), which regularizes the spectrum of the softmax input and prevents collapse of gradients (Ji et al., 30 Sep 2025).

3. Algorithmic Construction and Inference

3.1 RMNet Block Conversion

The conversion proceeds blockwise, expanding $C$-dim feature maps to $2C$ via direct assignment and identity-mapped splitting, modifying batch normalization statistics, and concatenating outputs prior to the final convolution. This process is exact for v1 blocks and is implemented with single-branch convolutions at inference.

3.2 Bubble-Enriched FEM

Each mesh element computes and stores enriched shape functions using spectrally accurate $p$th-order bases for the bubble correction, assembling the global system using only the enriched degrees of freedom. Static condensation may be used to eliminate element-internal bubbles for efficiency.

3.3 Skipless Transformer Training

Standard transformer block architectures are used verbatim, minus residual additions. Initialization ensures Jacobian conditioning is preserved without explicit skip connections. Empirically, AdamW or SOAP optimizers with proper initialization recover optimization stability and convergence parity with residual baselines (Ji et al., 30 Sep 2025).

4. Stability, Pruning, and Hierarchical Representation

A significant advantage of residual-free architectures is improved amenability to model sparsification and refined feature hierarchies:

• In RMNet, channel pruning is unimpeded by skip-branch dependencies, supporting up to 70–80% filter removal in a single feedforward chain (Meng et al., 2021).
• In skipless transformers, hierarchical abstraction is preserved, with each layer building solely upon its predecessor, leading to richer and more interpretable representations (as seen in PCA visualizations under DINO pretraining) (Ji et al., 30 Sep 2025).
• The residual-free bubble FEM achieves stability for advection-dominated problems even at Péclet numbers up to $10^{14}$, with exponential convergence with respect to the bubble polynomial order—substantially outperforming traditional $hp$-FEM under extreme advection (Kryven et al., 2016).

5. Empirical Evaluation and Trade-Offs

Benchmarks across domains validate the efficiency and accuracy of residual-free architectures. In image classification, RMNet variants surpass ResNet and RepVGG on both accuracy and throughput, with RMNet-152×6_32 achieving 80.4% Top-1 on ImageNet compared to 77.8% for ResNet-152, at higher pruning ratios and better speed-accuracy trade-offs. Skipless Transformers, with appropriate initialization, match or surpass their residual-based counterparts in both supervised (ImageNet-1k) and self-supervised (DINO, VOC2012, COCO) tasks, achieving, for example, 80.8% Top-1 accuracy (ViT-Base + SOAP, 300 epochs) versus 80.3% for the residual baseline (Ji et al., 30 Sep 2025).

Residual-free bubble-enriched FEM shows, for moderate $p$, absence of oscillations and high accuracy at extreme Péclet numbers, while reducing degrees of freedom per element relative to $hp$-FEM.

6. Broader Implications and Applications

Residual-free principles enable scaling to deeper architectures without “residual drift” or collapse, promote efficient high-ratio pruning, and simplify single-path inference for deployment on hardware accelerators. They also provide new grounding for the necessity (or irrelevance) of skip connections: with proper conditioning, residual-additions are not fundamentally required for efficient optimization or representational richness.

Applications extend from numerical simulation of advection-dominated PDEs to vision, segmentation, and object discovery tasks in high-dimensional machine learning. The methodologies are also compatible with existing computational accelerators, including FlashAttention kernels and channel-pruned CNNs, due to their use of standard architectural primitives and single-branch execution paths.

7. Outlook and Future Research Directions

Residual-free architectures are poised to impact both theoretical understanding and practical system design:

• Exploration of ultra-deep transformer architectures ($>$100 layers) without degradation due to representation “flattening.”
• Design of new module types (e.g., residual-free convolutional blocks and MLP-Mixers) to exploit single-branch benefits while retaining or enhancing expressiveness.
• Further innovations in initialization (e.g., per-head diagonal dominance, learnable spectral regularization) to generalize beyond current methods.
• Potential advances in adaptive mesh and bubble-enriched finite element strategies based on local error indicators rather than inherited residual correction.

A plausible implication is that, while residual connections historically eased the training of deep neural and numerical architectures, careful architectural reparameterization, analytical enrichment, or initialization can yield models that are empirically and theoretically robust, more pruning-friendly, and computationally streamlined across domains (Kryven et al., 2016, Meng et al., 2021, Ji et al., 30 Sep 2025).