mHC-lite: Exact Doubly Stochastic Residuals

• mHC-lite is an architectural modification for deep neural networks that guarantees exact doubly stochastic residual connections via convex combination reparameterization.
• It reparameterizes residual mixing matrices as convex combinations of permutation matrices, eliminating the need for iterative normalization and specialized kernels.
• Empirical results show mHC-lite improves gradient stability, throughput, and training reproducibility compared to traditional and mHC methods.

mHC-lite is an architectural modification for deep neural networks that guarantees exact doubly stochastic residual connections using a simple, non-iterative parameterization. It is motivated by limitations observed in prior frameworks—Hyper-Connections (HC) and Manifold-Constrained Hyper-Connections (mHC)—where dynamic residual mixing accelerates convergence but can destabilize training due to unconstrained or approximately normalized mixing matrices. mHC-lite achieves doubly stochasticity by reparameterizing these matrices as explicit convex combinations of permutation matrices, circumventing the need for iterative normalization and specialized kernel implementations, and delivering comparable or superior empirical performance to previous methods (Yang et al., 9 Jan 2026).

1. Background: HC, mHC, and Their Limitations

HC generalizes classical residual links by introducing a learnable, per-token residual mixing matrix $H^{\text{res}}_\ell \in \mathbb{R}^{n\times n}$ acting on $n$ simultaneous streams at each layer. This generalization accelerates convergence but lacks inherent norm or stability constraints, risking gradient explosion or instability as depth increases.

mHC projected $H^{\text{res}}_\ell$ onto the Birkhoff polytope $\mathcal{B}_n$ (the set of $n\times n$ doubly stochastic matrices) using $T = 20$ iterations of the Sinkhorn–Knopp (SK) algorithm, bounding $\| H \|_2 \leq 1$ and ensuring closure under matrix multiplication. However, mHC suffers from two critical issues:

• Approximation gap: A finite $T$ typically fails to ensure exact doubly stochasticity, particularly for ill-conditioned inputs (evidenced by ~28% of SK inputs with condition numbers $1/\nu \geq 10^{13}$), allowing instability to accumulate with depth.
• Engineering limitations: SK’s efficiency requires bespoke fused CUDA kernels, heightening engineering complexity and hindered portability (Yang et al., 9 Jan 2026).

2. Theoretical Foundation: Birkhoff–von Neumann Theorem

The mathematical basis for mHC-lite is the Birkhoff–von Neumann theorem, which states that every doubly stochastic matrix $X \in \mathcal{B}_n$ can be written as a convex combination of permutation matrices $P_1, \ldots, P_{n!}$:

$$X = \sum_{k=1}^{n!} a_k P_k, \quad a_k \geq 0, \quad \sum_{k=1}^{n!} a_k = 1.$$

HC and mHC acknowledge this but do not directly operationalize it; mHC-lite, in contrast, leverages this decomposition to generate exactly doubly stochastic matrices.

3. Mathematical Formulation and Block Architecture

mHC-lite adopts the overall block structure of HC/mHC, acting on residual streams $x_\ell \in \mathbb{R}^{n \times c}$:

1. Flatten and normalize: $$\hat{x}_\ell = \operatorname{RMSNorm}(\operatorname{reshape}(x_\ell, [1, nc]))$$
2. Pre/post scalars:
   • $$H^{\mathrm{pre}}_\ell = \sigma(\alpha^{\mathrm{pre}}_\ell \hat{x}_\ell W^{\mathrm{pre}}_\ell + b^{\mathrm{pre}}_\ell)$$
   • $$H^{\mathrm{post}}_\ell = 2\sigma(\alpha^{\mathrm{post}}_\ell \hat{x}_\ell W^{\mathrm{post}}_\ell + b^{\mathrm{post}}_\ell)$$
3. Permutation weighting:
   • $$a_\ell = \operatorname{softmax}(\alpha^{\mathrm{res}}_\ell \hat{x}_\ell W^{\mathrm{res}}_\ell + b^{\mathrm{res}}_\ell) \in \Delta^{n! - 1}$$
4. Residual mixing matrix:
   • $$H^{\mathrm{res}}_\ell = \sum_{k=1}^{n!} a_{\ell k} P_k$$

This guarantees $H^{\mathrm{res}}_\ell$ is doubly stochastic for all $\ell$ by construction. Only native pointwise nonlinearities and matrix multiplications are required, avoiding iterative or custom-kernel operations.

4. Algorithm and Implementation

The typical forward pass through an mHC-lite block is as follows:

hat_x_ell = RMSNorm(reshape(x_ell, [1, n*c])) H_pre = sigmoid(alpha_pre * hat_x_ell @ W_pre + b_pre) # [1 x n] H_post = 2 * sigmoid(alpha_post * hat_x_ell @ W_post + b_post) # [1 x n] logits_res = alpha_res * hat_x_ell @ W_res + b_res # [1 x n!] a_ell = softmax(logits_res) # [1 x n!] vec_H_res = a_ell @ Perm # [1 x n^2] H_res = reshape(vec_H_res, [n, n]) x_ell_plus_1 = H_res @ x_ell + H_post * f(H_pre * x_ell; W_main)

• All $n!$ permutation matrices $P_k$ are stored as a [n! $\times$ n²] constant matrix $\mathrm{Perm}$.
• The runtime complexity is dominated by a [1 $\times$ n!] $\times$ [n! $\times$ n²] multiplication and O($n!n^2$ + $n nc$) FLOPs per layer (for $n=4$ as in HC, this is only 384 multiplications, negligible relative to attention/MLP computation).

5. Empirical Evaluation

Extensive experimental results on OpenWebText and FineWeb-Edu demonstrate:

• Convergence and final loss: mHC-lite matches or slightly surpasses mHC across three scales (S/M/L). For example, on FineWeb-Edu (S-scale) train/val losses: HC 3.475/3.471, mHC 3.474/3.469, mHC-lite 3.471/3.467.
• Gradient stability: Both mHC and mHC-lite substantially reduce gradient fluctuations versus HC; mHC-lite further lowers gradient-norm variance and mean.
• Throughput: On 8xA100 GPUs (nanoGPT-M, 12-layer), mHC-lite achieves ~103% of HC throughput using naive implementation, while mHC yields ~92–93% of HC (PyTorch SK). Specialized mHC implementations reduce overhead but do not outperform mHC-lite in baseline conditions.
• Stability in depth: Empirical drift in column sums of $\prod_\ell H^{\text{res}}_\ell$ is up to +220% for mHC in a 24-layer net, while mHC-lite’s remain exactly at 1.

6. Engineering and Scalability Considerations

Advantages of mHC-lite include:

• Exact normalization: Elimination of any approximation gap inherent to finite-step SK.
• No custom kernels: All computations use native BLAS/cuBLAS operations.
• Drop-in portability: No requirement for bespoke kernel fusion or adaptation.
• Gradient norm control: Prevents gradient explosion from non-normalized residuals.

A principal limitation is the factorial growth of permutations, which bounds $n$ to small values ($n=4$ in HC/m yields 24 permutations). For larger $n$, subsampling or learning a subset of permutations must trade off expressivity against compute. The convex combination model may underparametrize structures achievable via unconstrained SK, yet experiments show comparable or superior practical results on standard benchmarks.

7. Broader Context and Significance

mHC-lite represents a practical instantiation of classical convex-analytic results within deep networks, advancing regime-stable, efficiently implementable residual mixing. It sidesteps both numerical and systems-level obstacles of SK-based projections, ensuring mathematically robust and expressively sufficient dynamics in current model deployments. The methodology dovetails with ongoing research on manifold-constrained parameterizations and controlled dynamical systems in deep learning (Yang et al., 9 Jan 2026). The released implementation supports reproducible research and adoption in large-scale sequence modeling pipelines.