When Does Sparsity Mitigate the Curse of Depth in LLMs

Abstract: Recent work has demonstrated the curse of depth in LLMs, where later layers contribute less to learning and representation than earlier layers. Such under-utilization is linked to the accumulated growth of variance in Pre-Layer Normalization, which can push deep blocks toward near-identity behavior. In this paper, we demonstrate that, sparsity, beyond enabling efficiency, acts as a regulator of variance propagation and thereby improves depth utilization. Our investigation covers two sources of sparsity: (i) implicit sparsity, which emerges from training and data conditions, including weight sparsity induced by weight decay and attention sparsity induced by long context inputs; and (ii) explicit sparsity, which is enforced by architectural design, including key/value-sharing sparsity in Grouped-Query Attention and expert-activation sparsity in Mixtureof-Experts. Our claim is thoroughly supported by controlled depth-scaling experiments and targeted layer effectiveness interventions. Across settings, we observe a consistent relationship: sparsity improves layer utilization by reducing output variance and promoting functional differentiation. We eventually distill our findings into a practical rule-of-thumb recipe for training deptheffective LLMs, yielding a notable 4.6% accuracy improvement on downstream tasks. Our results reveal sparsity, arising naturally from standard design choices, as a key yet previously overlooked mechanism for effective depth scaling in LLMs. Code is available at https://github.com/pUmpKin-Co/SparsityAndCoD.

• The paper demonstrates that sparsity, through both implicit training dynamics and explicit architectural enforcement, robustly regulates variance propagation to mitigate the curse of depth.
• It employs three metrics—causal, permutation, and usefulness scores—to systematically quantify layer effectiveness and reveal depth-induced redundancy.
• Empirical and theoretical analyses show that sparsity mechanisms like Grouped Query Attention and Mixture-of-Experts restore layer efficiency and improve overall LLM performance.

Sparsity as a Mechanism to Mitigate the Curse of Depth in LLMs

Introduction and Motivation

LLMs based on deep Transformer architectures have consistently demonstrated that, as network depth increases, the marginal utility of additional layers diminishes—a phenomenon dubbed the "curse of depth" (CoD). This manifests as severe under-utilization and functional redundancy in the deepest layers, notably in Pre-LN Transformers, where output variance accumulates sub-exponentially with depth. The resulting drift of layer Jacobians toward the identity drastically impairs the ability of deep blocks to implement nontrivial transformations. Traditionally, variance growth has been controlled through specialized initializations, normalization techniques, or modifications to the residual pathway. However, this work asserts a fundamentally different modulation mechanism: sparsity, whether implicit (arising from training dynamics) or explicit (architecturally enforced), operates as a robust regulator of variance propagation—directly addressing the CoD without requiring disruptive architectural changes.

Formalization of the Curse of Depth

Empirical investigations were conducted to verify dominant trends underpinning the CoD. Models were scaled from 12 to 32 layers with parameter count and other hyperparameters controlled. Three orthogonal metrics assessed layer effectiveness:

• Causal Score: Quantifies the influence of each layer on downstream transformations, via representation drift caused by layer removal.
• Permutation Score: Measures specialization by the performance penalty from swapping layer positions; low values indicate redundancy.
• Usefulness Score: Estimates the nonlinearity contributed by a layer, based on the performance change from its linearization.

All metrics strongly degrade with depth escalation—e.g., usefulness drops from 0.75 (at 12 layers) to 0.53 (at 32 layers)—with the majority of layers in the deepest models functionally idling despite substantial parameter and compute expense. Output variance and the norm of $(J-I)$ (Jacobian relative to identity) similarly demonstrate depth-induced drift toward ineffective, near-linear transformations.

Figure 1: Last-layer variance increases sharply with model depth, confirming residual stream accumulation and diminishing marginal utility of additional layers.

Theoretical Framework: Sparsity as a Variance Regularizer

A critical theoretical contribution precisely quantifies the effect of sparsity on variance propagation. Given a residual recursion,

$r_{\ell+1} = r_\ell + W_\ell(D_\ell r_\ell)$

where $D_\ell$ is a diagonal mask parameterizing sparsity and $W_\ell$ is a random transformation, per-layer variance gain is proportional to the density $\rho_\ell$ of $D_\ell$. Explicitly,

$$\mathrm{Var}(r_L) \le \mathrm{Var}(r_0)\prod_{\ell=0}^{L-1}(1+\sqrt{\alpha_\ell\rho_\ell})^2$$

for $\alpha_\ell$ a bound on the second moment of $W_\ell$. Thus, both explicit and emergent patterns that drive $\rho_\ell$ lower monotonically reduce aggregate variance growth across depth. Notably, when sparsity is induced through training (implicit), weights and sparsity become statistically coupled, but the variance suppression effect persists.

Empirical Analysis of Sparsity Regimes

The paper systematically dissects both implicit (dynamically induced) and explicit (architecture-enforced) sparsity modalities.

Implicit Sparsity

Weight Decay: Increasing the $L_2$ penalty $\lambda$ fosters a greater fraction of (effectively) zero-weight parameters. Variance in the terminal layer decreases monotonically with $\lambda$ up to a regime, after which over-regularization impairs capacity.

Sequence Length Scaling: Longer context tokens drive sharper attention distributions due to positional bias and softmax normalization, increasing attention matrix sparsity. This sparsifies attention maps and attenuates stochasticity, reducing variance per coordinate via the law of large numbers.

Figure 2: Average attention maps for head 0 indicate concentration of attention with extended sequence length, visualizing the emergence of sparsity patterns in the attention matrices.

Explicit Sparsity

Grouped Query Attention (GQA): Partitioning query heads into groups with shared key/value projections reduces the number of independent key/value computations, enforcing a fixed sparsity pattern. This averaging effect, under standard statistical assumptions, provides an explicit $1/G$ reduction in variance, where $G$ is the number of groups.

Figure 3: Output variance decreases as GQA group size increases, evidencing explicit architectural sparsity as a variance suppressor.

Mixture-of-Experts (MoE): Gating activates only $k$ experts from a pool of $E$, producing an explicit average that reduces both output and Jacobian variance by $1/k$ under independence. MoE architectures were found to deliver approximately $3$–$6\times$ variance attenuation compared to dense baselines, substantially increasing both perplexity performance and layer usefulness.

Figure 4: MoE variant with 400M activated parameters exhibits substantially reduced last-layer variance compared to the dense control, exemplifying the averaging benefit of the Top-$k$ sparsity mechanism.

Interplay Between Sparsity and Depth Utilization

Ablation studies reveal that naive depth scaling (e.g., from $L=16$ to $L=32$ at constant parameter count) paradoxically reduces both accuracy and layer effectiveness, confirming the CoD. However, integrating sparsity mechanisms reverses this trend. Extending context, introducing moderate weight decay, employing GQA, or leveraging MoE all contribute to improved final accuracy and substantial restoration (or even improvement) of the usefulness score. The cumulative effect of multiple sparsity strategies fosters models that outperform shallow baselines both in accuracy and depth efficiency.

Figure 5: Progressive performance gains when scaling the 1.2B model to depth $L=32$ via stacking with various ``sparsity'' modules, demonstrating the indispensability of sparsity for effective deep scaling.

Practical and Theoretical Implications

The evidence presented replaces the conventional optimization-centric perspective on sparsity (as a tool for computational savings) with a more fundamental viewpoint: sparsity operates as an inherent variance regulator. In both implicit and explicit forms, it enables Transformers to exploit additional depth by constraining variance escalation, promoting specialization and representational diversity in deeper blocks. This recontextualization suggests that architectural and training decisions (e.g., selection of attention mechanisms, regularization schemes, expert routing in MoE) should be evaluated not just by compute efficiency, but as levers controlling functional network depth and mitigating the CoD.

On the theoretical front, these results elucidate the mechanisms underlying why certain architectural trends—such as the dominance of MoE, popularity of grouped and multi-query attention, or the priority placed on supporting long contexts—have proven so effective for the current generation of LLMs. Properly tuned sparsity emerges as a necessary condition for efficient utilization of ever-increasing depth, likely essential for advancing toward larger, deeper, and more cost-effective LLMs.

Conclusion

This work provides comprehensive evidence—both theoretical and empirical—that sparsity, whether induced implicitly or enforced explicitly, acts as a universal regularizer of variance propagation in deep LLMs, directly counteracting the curse of depth. Sparsity modulates variance growth, thus restoring the utility of deep transformations and enabling deep scaling to translate into functional capacity and improved downstream accuracy. These insights reframe architectural and training design, underscoring sparsity as a central consideration for future advances in LLM depth scaling and overall efficiency.