Low-Rank & Sparse Recurrent Connectivity

• Low-rank and sparse recurrent connectivity is a design paradigm that factorizes RNN weight matrices into low-rank components with sparse masks, reducing parameters and enhancing interpretability.
• This approach precisely controls spectral properties to modulate memory horizons and stability, yielding compact models with robust performance under distribution shifts.
• Practical integration into architectures like GRUs and LSTMs demonstrates improved data efficiency and mechanistic clarity in tasks such as modular addition and closed-loop control.

Low-rank and sparse recurrent connectivity refers to recurrent neural network (RNN) architectures in which the recurrent weight matrices are explicitly parameterized or learned to be both low-rank and sparse, yielding compact, memory-efficient models with interpretable and controllable dynamical properties. This structural constraint modulates spectral and dynamical properties of the recurrent weights, underpins mechanistic interpretability in certain algorithmic tasks, and facilitates robust generalization under distribution shift in closed-loop control scenarios. The approach is applied both as a structural prior for model design and as an emergent property in tasks amenable to mechanistic probe, such as modular addition via Fourier multiplication.

1. Mathematical Formulation of Low-Rank and Sparse Recurrent Connectivity

Let $W_{\mathrm{rec}} \in \mathbb{R}^{h \times h}$ denote the recurrent weight matrix in a standard RNN, LSTM, GRU, or related passthrough block. The low-rank and sparse parameterization decomposes this matrix as follows:

$W_{\mathrm{rec}}(r,s) = W_1(r)\, W_2(r) \odot M(s)$

where:

• $W_1(r) = U_r \Sigma_r^{1/2}$, $W_2(r) = \Sigma_r^{1/2} V_r^T$, with $U_r,\Sigma_r,V_r$ from the rank-$r$ truncated singular value decomposition (SVD) of $W_{\mathrm{rec}}$ (Tumma et al., 2023).
• $M(s) \in \{0,1\}^{h \times h}$ is a fixed random mask with i.i.d. entries $M_{ij} \sim 1 - \mathrm{Bernoulli}(s)$, zeroing out a fraction $s$ of elements at initialization.
• The effective parameter count is reduced from $h^2$ to $2hr$ for the pure low-rank case or $2hr + h$ if an additional diagonal is included (Barone, 2016).

In tasks focusing on mechanistic interpretability, such as modular addition, the emergent RNN weights are observed to have low effective rank under SVD: for instance, a $256 \times 256$ $W_{\mathrm{rec}}$ is empirically reduced to a rank-32 approximation that preserves nearly 90% of its spectral energy and full task accuracy (Rangamani, 28 Mar 2025).

2. Spectral Properties and Their Dynamical Implications

The rank ($r$) and sparsity ($s$) parameters interact with initialization schemes (e.g., Glorot-uniform/GU-spec or orthogonal/ortho-spec) to tightly control the spectrum of $W_{\mathrm{rec}}$ (Tumma et al., 2023). Theoretical results in the large-$h$ limit include:

• Under GU-spec: The spectral radius $\rho[W_{\mathrm{rec}}(r,s)]$ and spectral norm $\|W_{\mathrm{rec}}(r,s)\|_2$ both decrease monotonically with increasing $s$; norm is largely independent of $r$.
• Under ortho-spec: The spectral radius increases with $r$; the spectral norm remains fixed at unity irrespective of $r$.

Empirically, increasing $s$ (more zeros) contracts dynamical range and shortens memory horizon by flattening singular value decay; lowering $r$ steepens spectrum decay, effectively reducing system dimensionality. Such control aligns the recurrence dynamics with the demands of specific modeling regimes, such as closed-form continuous-time neural networks (CfCs), where robustness under distributional shift and well-defined memory horizons are required (Tumma et al., 2023).

3. Integration into Neural Architectures and Training Dynamics

Low-rank and sparse parameterizations are modular and integrate directly into passthrough network variants including GRUs, LSTMs, ResNets, and Highway Networks (Barone, 2016). In practice:

• Each gate or transform matrix $W$ is factorized as $UV$ or $UV + D$ (with a trainable diagonal $D$) to preserve local coordinate effects otherwise suppressed in purely low-rank models.
• Computational savings are substantial: parameter counts drop from $n^2$ (full-rank) to $2nr$, supporting large hidden states on limited memory budgets. Table 1 summarizes the scaling for $n=512, r=32$:

Model	Parameters
Full-rank	262,144
Low-rank ($r$)	32,768
Low-rank+diag	33,280

• Training typically requires initialization normalization (e.g., weight normalization on $U, V$), row-norm clipping, and may benefit from starting with pure low-rank and switching to "+diag" if convergence is slow or expressivity is inadequate (Barone, 2016).

4. Mechanistic Interpretability: Fourier Sparsity in Algorithmic Tasks

In tasks such as modular addition, trained RNNs exhibit both low-rank and extreme sparsity in the Fourier domain (Rangamani, 28 Mar 2025). Specifically, after Fourier decomposition of embeddings, hidden states, and unembedding weights, only a small number (six) of frequency components carry significant energy across all layers. The representations are thus supported on a 12-dimensional subspace (six cosine-sine pairs), and the network's internal computation implements exact arithmetic Fourier multiplication:

$$\cos(\omega_k a) \cos(\omega_k b) - \sin(\omega_k a) \sin(\omega_k b) = \cos[\omega_k(a+b)]$$

Empirical ablation demonstrates that removing singular frequencies degrades performance gradually, with accuracy dropping to chance after all informative components are eliminated. This demonstrates causality between low-rank/Fourier sparsity and task solution (Rangamani, 28 Mar 2025).

5. Empirical Performance and Robustness in Sequential and Control Tasks

Empirical evaluations show that low-rank and sparse recurrent structures yield substantial practical advantages. On standard synthetic and real-world tasks (Tumma et al., 2023, Barone, 2016):

• In closed-loop, out-of-distribution (OOD) regimes, CfCs with moderate low-rank ($r=5\text{--}16$), mild sparsity ($s \approx 0.0\text{--}0.2$), and orders-of-magnitude fewer parameters outperform or match full, dense RNNs and LSTMs. For example, in Seaquest under OOD perturbation, a rank-5, $s = 0.2$ CfC achieves normalized reward $1.26 \pm 0.03$ with 640 parameters, compared to the full model's $0.95 \pm 0.03$ with 4096 parameters.
• On permuted MNIST, low-rank + diagonal GRU (state 128, rank 24) matches full-rank test accuracy (93.5% vs. 92.8%) with less than half the parameters.
• In language modeling, low-rank + diagonal GRU with one-sixth the parameters matches or improves upon standard full GRU baseline perplexity.
• On addition and memory-copy synthetic tasks, adding a diagonal term resolves expressivity limitations of pure low-rank, supporting successful training in challenging regimes (Barone, 2016).

6. Hyperparameter and Architectural Guidelines

Guidelines derived from empirical and theoretical analysis (Tumma et al., 2023, Barone, 2016):

• Moderate rank ($r \approx n/8$ to $n/16$ or as needed by spectral decay) offers good tradeoff of memory capacity and regularization.
• Mild sparsity ($s \lesssim 0.2$) preserves robustness and capacity; very high sparsity ($s \geq 0.5$) impairs expressivity and performance.
• Use weight normalization and row-norm clipping for numerical stability.
• Switch from pure low-rank to "+diag" if required by task or convergence.
• Combine with modern regularization strategies (dropout, layer norm) for further enhancement.
• Structural constraints are orthogonal to other modeling advances and compose with skip connections, memory-augmented architectures, and beyond.

7. Significance and Broader Context

Low-rank and sparse recurrent connectivity provides a principled dynamical prior modulating the memory horizon, local Lipschitz sensitivity, and effective activity dimensionality in recurrent architectures. Empirical results demonstrate that such parameterizations induce strong regularization, promote interpretability in mechanistic regimes, and yield substantial improvements in data efficiency, robustness, and stability under distributional shifts. A plausible implication is that further mechanistic insights and architectural variants—such as structured banded, convolutional, or tensor decompositions—could extend this framework to broader classes of sequential modeling problems (Tumma et al., 2023, Rangamani, 28 Mar 2025, Barone, 2016).