Activation Decomposition Methods

• Activation Decomposition Methods are techniques that transform neural activations into lower-dimensional, statistically structured components using methods like SVD and HOSVD.
• They enable improved model interpretability by isolating decisive features and generating disentangled saliency maps that clearly delineate significant activation subspaces.
• These methods support efficient model compression and robust OOD detection by distinguishing between key and residual activation components, leading to measurable performance gains.

Activation decomposition methods refer to a diverse set of techniques that transform, compress, or analyze intermediate neural network activations by representing them as combinations of lower-dimensional or statistically-structured components. These methods span applications from model interpretability and out-of-distribution (OOD) detection, to model quantization, compression, and improved optimization. Central to this field is the use of Singular Value Decomposition (SVD), High-Order SVD (HOSVD), or related decompositions to partition or compress activations, with specific approaches tailored to task structure and constraints. This article reviews principal methodologies, theoretical foundations, and empirical results, as documented across recent literature.

1. Mathematical Formulations of Activation Decomposition

Activation decomposition commonly employs linear algebraic techniques to represent neural activations as the sum or projection into subspaces defined by leading singular vectors or factors. Let $A$ denote an activation matrix (or unfolded tensor):

• Matrix SVD: $A = U \Sigma V^\top$, where $U$ and $V$ are orthonormal matrices, and $\Sigma$ is diagonal with descending singular values. Truncating to $K$ components yields $\tilde{A} = U_{(K)} \Sigma_{(K)} V_{(K)}^\top$ (Nguyen et al., 2024).
• Tensor HOSVD: For $N$-way activation tensor $$\mathcal{X} \in \mathbb{R}^{M_1 \times \dots \times M_N}$$, $$\mathcal{X} = \mathcal{S} \times_1 U^{(1)} \times_2 \dots \times_N U^{(N)}$$, with factor matrices $U^{(n)}$ as left singular vectors of mode-$n$ unfoldings, and $\mathcal{S}$ the core tensor (Nguyen et al., 2024).
• Activation Subspace Projections: In classification, the network head weights $W \in \mathbb{R}^{c \times n}$ are decomposed via $W = U \Sigma V^\top$. The columns of $V$ define “decisive” ($V_k$) and “insignificant” ($V_{-k}$) right-singular subspaces. For any activation $a \in \mathbb{R}^n$, $$a = a_\mathrm{dec} + a_\mathrm{ins} = P_\mathrm{dec} a + P_\mathrm{ins} a$$, where $P_\mathrm{dec} = V_k V_k^\top$, $P_\mathrm{ins} = V_{-k} V_{-k}^\top$, and $P_\mathrm{dec} + P_\mathrm{ins} = VV^\top$ (Zöngür et al., 29 Aug 2025).

Such decompositions are adopted both for runtime compression and for analytical separation based on model semantics, dynamic range, or statistical structure.

2. Decomposition for Model Interpretability

Activation decomposition provides a foundation for interpretable neural network analysis, particularly via decomposition-enhanced Class Activation Map (CAM) variants:

• Decom-CAM: Saliency tensors $S_k$ at layer $k$ are constructed as $S_k = (\partial y_\ell/\partial A_k) \odot A_k$, then flattened and SVD-decomposed. Top $p$ singular vectors span orthogonal directions, yielding $\{F_{k,i}\}_{i=1}^p$ as disentangled feature-level saliency maps. Integration via importance weights from occlusion-based class impact forms the final map $S(x,y)$ (Yang et al., 2023).
• DecomCAM: Class-discriminative maps from top-$P$ channels are stacked, SVD-decomposed, and projected to $Q$ orthogonal sub-saliency maps (OSSMs), $\{H_q\}$, each aligning with semantically distinct image regions. Final attribution scores combine OSSMs using causal impact-based softmax weights (Yang et al., 2024).

Empirically, these methods deliver increased localization, feature granularity (e.g., object part assignment), and robustness across varying model confidence bins, surpassing standard Grad-CAM and CAM++ in deletion/insertion and Pointing Game metrics.

3. Decomposition for Out-of-Distribution Detection

Activation decomposition is exploited for subspace-based OOD scoring. In ActSub (Zöngür et al., 29 Aug 2025):

• Decisive directions ($P_\mathrm{dec}$) are the activation modes most shaped by classification, offering robust ID vs. Near-OOD separation via energy-based scoring on shaped logits.
• Insignificant directions ($P_\mathrm{ins}$) are “softmax-invariant” or near the classification head’s nullspace; these modes are under-constrained by supervision and retain generic features, providing high discrimination for Far-OOD via cosine-similarity-based scores.

Combined scoring ($$S_\mathrm{final} = S_\mathrm{ins}^\lambda \cdot S_\mathrm{dec}$$) yields state-of-the-art AUROC and FPR improvements on ImageNet and CIFAR-10 OOD benchmarks, demonstrating a statistically significant uplift over prior activation shaping detectors.

Method	Near-OOD AUC/FPR	Far-OOD AUC/FPR
SCALE (baseline)	81.36% / 59.76%	96.53% / 16.53%
ActSub w/ SCALE	84.24% / 52.60%	96.96% / 14.29%

4. Activation Decomposition in Model Compression and Quantization

Low-rank and activation-aware decomposition techniques are instrumental in reducing the footprint of both activations and model parameters:

• Activation Map Compression: Truncated SVD and HOSVD approximate activations in convolutional and transformer models, yielding exact or near-exact gradient recovery up to the controlled truncation bias. HOSVD, which decomposes along batch, channel, height, and width, enables memory savings (e.g., 0.73 KB for MCUNet→CIFAR-10 at $ε=0.8$ vs 61 KB vanilla), while maintaining accuracy within $\sim1\%$ of the uncompressed baseline. Backward passes are up to $10$-$20\times$ faster, with theoretical and empirical convergence guarantees (Nguyen et al., 2024).
• QUAD for LLM Quantization: SVD projects activations into top-$k$ outlier directions and a residual subspace. Outliers are stored and processed in full precision, while the residuals are quantized at 4–8 bit precision. Offline calibration via SVD ensures coverage of heavy-tailed components, and parameter-efficient fine-tuning adapts the full-precision outlier weights, restoring original model accuracy (up to $98.5\%$–$100.6\%$ FP16 baseline) under high compression (Hu et al., 25 Mar 2025).
• NSVD for Weight Matrix Compression: Nested activation-aware SVD applies a whitening/rotation $T$ constructed from calibration set activations, followed by two-stage low-rank factorization—one rank-$k_1$ step tuned to $A T$ for minimal activation-aware loss, one rank-$k_2$ step for residuals. This paradigm yields tighter error bounds and up to $40$–$60\%$ perplexity reduction at high compression ratios in LLMs across diverse data domains (Lu et al., 21 Mar 2025).

Model (LLaMA-7B, 30% rank)	Baseline Perplexity	ASVD Best	NSVD
English sets	ref	-7–12%	-7–12%
Chinese/Japanese sets	ref	-16–55%	-16–55%

5. Decomposition-Enabled Adaptive/Hybrid Nonlinearity Design

Decomposition can also be applied at the functional level, segmenting and pairing activation functions to mitigate training pathologies:

• High-Dimensional Function Graph Decomposition (HD-FGD): Splits a complex activation $f$ into $K$ parallel terms $f(x) \approx \sum_{i=1}^K \phi_i(W_i x)$, each $\phi_i$ a simple nonlinearity acting on a projected subspace. For gradient stabilization, adversarial activations $\xi$ are constructed by integrating the reciprocal of each $\phi_i'$ (i.e., $$\partial \xi / \partial \chi_i = (\phi_i'(\chi_i))^{-1}$$). Alternating original and adversarial activations layer-wise reduces internal covariate shift and gradient deviation, yielding substantial improvements in convergence and accuracy across ResNets, Vision Transformers, and Swin-Tiny architectures (e.g., $+18$ to $+101$\% gains with adversarial pairing on Sigmoid/Tanh baselines) (Su et al., 2024).

Implementation is plug-and-play, requiring no macroarchitectural changes and scaling efficiently with $K\leq4$.

6. Multi-Activation and Domain Decomposition in Scientific Deep Learning

Activation decomposition principles extend to domain decomposition for PDE-solving neural networks. In Multi-Activation Function (MAF) approaches (Zhai, 20 Dec 2025):

• Subdomain-specific networks are joined via interface conditions, with internal representations blending global (e.g., tanh) and localized (Gaussian) activations as

$$\sigma^{[l]}(z) = \omega_1(x)\tanh(z_1) + \omega_2(x)\exp(-\|z_2\|^2/\gamma)$$

where $\omega_2(x)$ is strong near interfaces and decays with distance. This adaptively reallocates modeling capacity near regions of coefficient discontinuity, enabling up to $10^3\times$ tighter error in elliptic/parabolic interface problems relative to other PINN or decomposition-based competitors.

Theoretical generalization error bounds are established, tying solution accuracy to training loss and quadrature error.

7. Common Principles, Advantages, and Limitations

Activation decomposition methods rely on the statistical, geometric, or functional splitting of activation space to optimize computational efficiency, interpretability, or downstream task robustness. Across methodologies:

• Advantages:
  • Statistical separation (e.g., outlier suppression) underpins quantization and post-training compression without accuracy loss (Hu et al., 25 Mar 2025, Lu et al., 21 Mar 2025).
  • Decomposition enhances causal interpretability by isolating orthogonal, part-aligned features (Yang et al., 2023, Yang et al., 2024).
  • Subspace-specific scoring empowers OOD detection in both Near- and Far-OOD regimes (Zöngür et al., 29 Aug 2025).
  • Adaptive, spatially- or functionally-aware decompositions boost expressiveness and error control in scientific ML (Zhai, 20 Dec 2025, Su et al., 2024).
• Limitations:
  • SVD/HOSVD computational cost, though often amortized/offline, may be significant for extremely high-dimensional activations or tensors.
  • Effectiveness depends on alignment between calibration/activation statistics and deployment distributions, especially for compression or quantization (Lu et al., 21 Mar 2025).
  • Component selection (e.g., number of retained singular vectors) requires principled tuning to balance trade-offs in fidelity, interpretability, and efficiency.

These methods continue to underpin advances across model analysis, deployment, and understanding, with ongoing research addressing richer decompositions and further integration with downstream tasks.