Adaptive Collaborative Error Suppression

• Adaptive Collaborative Error Suppression is a self-adaptive mechanism that mitigates error accumulation in low-rank SVD-based compression of deep language models.
• It jointly optimizes intra-layer reconstruction and inter-layer error compensation using closed-form solutions derived from second-order activation statistics.
• ACES, integral to the SAES-SVD framework, consistently enhances post-compression perplexity and accuracy without relying on fine-tuning.

Adaptive Collaborative Error Suppression (ACES) is a mechanism for self-adaptive error mitigation within low-rank compression frameworks for deep models, particularly targeting the propagation of reconstruction errors in singular value decomposition (SVD)-based compression of LLMs. ACES operates by jointly optimizing both intra-layer reconstruction and inter-layer error compensation, thereby mitigating the tendency for local layerwise errors to accumulate and amplify downstream, a phenomenon not addressed by conventional per-layer SVD truncation. The method is typically instantiated as a component in frameworks such as the Self-Adaptive Error-Suppression SVD (SAES-SVD), which couples cumulative error-aware objectives with adaptive cross-layer weighting (Hu et al., 3 Feb 2026).

1. Motivation and Problem Formulation

In large-scale neural network compression, standard low-rank SVD-based methods compress each layer independently, focusing solely on minimizing the local (per-layer) reconstruction error between the original and compressed weights. However, this layerwise independence neglects the interaction between modules; the compression error in one layer alters the activations feeding subsequent layers, resulting in systematic bias and error accumulation. This effect is especially pronounced in transformer-based LLMs with deep, stacked architectures, where global output deviations from the full-precision baseline can be dominated by the compounded local errors.

ACES was introduced to address this critical limitation by adaptively coordinating the error suppression process across layers, ensuring that the cumulative deviation from the original model is explicitly minimized, rather than minimized only in a local, layerwise sense (Hu et al., 3 Feb 2026).

2. Mathematical Foundations and Objective

Let $W_\ell\in\mathbb{R}^{m\times n}$ denote the full-precision weights of layer $\ell$. Given a calibration batch, let $X_\ell\in\mathbb{R}^{n\times B}$ be the compressed input activations and $X_\ell^{\mathrm{fp}}\in\mathbb{R}^{n\times B}$ be the reference activations from the full-precision model. The standard local objective $$\min_{A_\ell,B_\ell}\| (A_\ell B_\ell - W_\ell) X_\ell \|_F^2$$ is augmented in ACES to incorporate an explicit alignment with the full-precision outputs $W_\ell X_\ell^{\mathrm{fp}}$, weighted by a tunable parameter $\alpha_\ell > 0$:

$$\min_{A_\ell,B_\ell} \| (A_\ell B_\ell - W_\ell) X_\ell \|_F^2 + \alpha_\ell \| (A_\ell B_\ell - W_\ell) X_\ell^{\mathrm{fp}} \|_F^2$$

This joint objective simultaneously suppresses the local error on compressed activations and the deviation from the ideal full-precision outputs, thereby targeting both local and accumulated errors. The minimization admits a closed-form rank-$r$ solution via truncated SVD, after reformulating the objective to depend only on second-order activation statistics (covariances), thus scaling to deep models (Hu et al., 3 Feb 2026).

3. ACES Adaptive Coefficient Selection

The core feature of ACES is its adaptive, layer-wise determination of the mixing coefficient ($\alpha_\ell$ or reparametrized as $\beta_\ell = \alpha_\ell/(1+\alpha_\ell)\in[0,1]$). Rather than setting $\alpha_\ell$ heuristically, ACES optimizes its value to maximize the concentration of spectral energy in the leading $r$ singular values after compression, subject to the fixed-rank constraint. This is formalized by maximizing the retained-energy ratio (RER):

$$\mathrm{RER}(\beta) = \frac{\sum_{i=1}^r \sigma_i(G_\ell(\beta))^2}{\|G_\ell(\beta)\|_F^2}$$

where $G_\ell(\beta) = S_\ell + \beta D_\ell$, with

• $S_\ell = W_\ell H_\ell^{1/2}$
• $D_\ell = W_\ell E_\ell H_\ell^{-1/2}$
• $H_\ell, E_\ell$ are input and cross covariance matrices.

ACES chooses $$\beta_\ell^* = \arg\max_{\beta\in[0,1]}\mathrm{RER}(\beta)$$. To avoid recomputing singular value decompositions for each candidate $\beta$, a closed-form quadratic surrogate for the numerator and denominator is derived by projecting onto the orthogonal complement of the leading singular subspace. All real roots in $[0,1]$ are obtained from the stationary condition, and the minimizer is chosen. This enables a one-shot, $O(1)$ procedure per layer to select $\beta_\ell^*$ (Hu et al., 3 Feb 2026).

4. Algorithmic Workflow

The ACES-driven compression proceeds in two principal passes over the model:

1. Statistic Collection: For each layer, activation covariances $H_\ell$ and error covariances $E_\ell$ are estimated from a calibration set by running the model in "parallel" full-precision and compressed modes.
2. Compression with ACES: For each layer:
   • The whitening operator $L_\ell = (H_\ell + \varepsilon I)^{-1/2}$ is constructed.
   • $S_\ell$ and $D_\ell$ are computed as above.
   • ACES determines $\beta_\ell^*$ via the closed-form maximization of the RER.
   • The effective target $G_\ell$ is formed and truncated SVD yields the rank-$r$ factors $A_\ell, B_\ell$.
   • The layer weights are replaced with $(A_\ell, B_\ell)$.

No layerwise fine-tuning or mixed-rank heuristics are involved. Inference replaces all full-precision weights with the compressed factors (Hu et al., 3 Feb 2026).

5. Theoretical Guarantees and Efficiency

ACES inherits the closed-form SVD optimality of the Eckart–Young theorem for the joint error objective defined by the CEALC–ACES formulation. The adaptive $\beta_\ell^*$ ensures that, for a given rank budget, the compressed layer's spectral energy is maximally concentrated in the top-$r$ singular values, thereby effectively utilizing the available representation capacity. All relevant quantities are computed in a single SVD per layer and an $O(r)$ coefficient solve, with overall memory/compute overhead limited to one pass per layer. Unlike fine-tuning-based methods, ACES does not alter the weights via gradient descent and instead relies exclusively on second-order activation statistics and SVD closed forms (Hu et al., 3 Feb 2026).

6. Empirical Performance and Significance

Extensive LLM compression evaluations, including on LLaMA-1/2/3 (7B/13B/30B) and Qwen2.5 (7B/32B), demonstrate that ACES, as part of SAES-SVD, yields consistent improvements in post-compression perplexity and zero-shot accuracy, without requiring fine-tuning or mixed-rank allocations:

• At 20% rank on LLaMA-7B, ACES improves WikiPPL from 7.95 (SVD-LLM baseline) to 7.17 and boosts zero-shot accuracy from 0.47 to 0.50, outperforming previous methods by reducing accuracy drop by ≈58% and narrowing PPL gap by ≈35%.
• At larger scales (LLaMA-13B and Qwen2.5-32B at 20% rank), perplexity reductions and 4–10 pt accuracy gains relative to competing SVD variants are reported.
• Ablation results indicate CEALC alone accounts for 15% PPL gain and 3 point accuracy increase, with ACES providing an additional ≈5% PPL and 1 pt accuracy benefit.
• Compression is post-hoc, requiring only a single pass of SVD per layer (≪1hr for LLaMA-7B), with memory footprint and inference latency reduced by up to 4× at high compression ratios (Hu et al., 3 Feb 2026).

7. Broader Context and Limitations

ACES, while implemented specifically for SVD-based LLM compression in (Hu et al., 3 Feb 2026), represents a broader methodological advance in adapting local error objectives in deep linear/compositional systems to explicitly account for cross-layer error propagation. By making the rank truncation process adaptive and collaborative—hence "Adaptive Collaborative Error Suppression"—it sidesteps the primary bottleneck of local error accumulation that limits the effectiveness of independent layerwise compression. A plausible implication is that similar adaptive, error-aware weighting strategies could generalize to other forms of multi-stage model compression or to tasks involving tensor factorizations and iterative sketching pipelines, provided suitable covariance statistics are accessible.

The method's primary limitation is its reliance on offline calibration data to estimate input covariance statistics, and it may be less effective if second-order statistics are not representative or if layer-nonlinearities interact strongly with the compression-induced drift. However, empirical robustness to $\beta_\ell$ selection and task variety suggests these constraints are not prohibitive in practice (Hu et al., 3 Feb 2026).

---

References:

• "SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compression" (Hu et al., 3 Feb 2026).