Linear equivalence of nonlinear recurrent neural networks

Abstract: Large nonlinear recurrent neural networks with random couplings generate high-dimensional, potentially chaotic activity whose structure is of interest in neuroscience, machine learning, ecology, and other fields. A fundamental object encoding the collective structure of this activity is the $N \times N$ covariance matrix. Prior analytical work on the covariance matrix has been limited to low-dimensional summary statistics, not the full high-dimensional object for a specific realization of the couplings. Recent work proposed an ansatz in which, at large $N$, the covariance matrix for a typical quenched realization takes the same form as that of a linear network with the same couplings, driven by independent noise, with mean-field order parameters setting the effective transfer function and the noise spectrum. Here, we derive this ansatz using the two-site cavity method, providing two distinct derivations that offer complementary perspectives. The first decomposes each unit's activity into a linear component and a nonlinear residual, and shows that cross-covariances between residuals at distinct sites are strongly suppressed, so that residuals act as independent noise within a linear network. The second writes a self-consistent matrix equation for the covariance matrix. A naive Gaussian closure for the joint statistics of activity at distinct sites gives the wrong equation; the cavity method separates Gaussian and non-Gaussian contributions, which enter at the same order, and produces the correct one. We verify the predictions numerically across a range of network sizes. These results extend linear equivalence from feedforward high-dimensional nonlinear systems, where the weights being analyzed are independent of their inputs, to recurrent networks, where the activities depend on the same couplings that generate them.

• The paper shows that the nonlinear RNN covariance matrix converges element-wise to that of a linear system with i.i.d. noise.
• It employs dual analytical methods—linear-residual decomposition and a two-site cavity approach—to derive controlled subleading corrections.
• Numerical simulations confirm that off-diagonal errors decay as 1/N and as 1/√N, validating the precise spectral equivalence.

Linear Equivalence in Nonlinear Recurrent Neural Networks

Introduction

The paper "Linear equivalence of nonlinear recurrent neural networks" (2604.23489) establishes that, for large-scale nonlinear recurrent neural networks (RNNs) with random couplings, the high-dimensional stationary-state covariance matrix is itself equivalent, up to controlled subleading corrections, to that of a corresponding linear network with identical couplings subjected to independent noise. This result bridges a longstanding gap between analytic tractability in linear systems and the complexity of nonlinear RNNs, with implications for the statistical mechanics of high-dimensional disordered systems in neuroscience, machine learning, and beyond.

Problem Setting and Analytical Gap

Random nonlinear RNNs exhibit complex, often chaotic, dynamics, with their activity encoded by an $N \times N$ covariance matrix. For linear systems with random connectivity, this object is tractable via direct linear algebra and random matrix theory. In contrast, nonlinear cases have only been understood through low-dimensional order parameters yielded by dynamical mean-field theory (DMFT), leaving the realization-dependent covariance matrix analytically inaccessible. Recent conjecture by Shen and Hu asserts that, in the large-$N$ limit, the covariance structure of the nonlinear system coincides with its linearization when operated at self-consistently determined noise and gain.

Main Results

The paper provides two independent derivations of this "linear equivalence" using the two-site cavity method, establishing:

• For i.i.d. random couplings, the full covariance matrix in the nonlinear network matches the corresponding linear network to element-wise precision $O(1/\sqrt{N})$.
• Off-diagonal elements, which characterize cross-unit synchrony and are typically $O(1/\sqrt{N})$, are reproduced up to errors $O(1/N)$.
• The effective parameters for the equivalent linear system (transfer function and noise spectrum) are rigorously connected to the DMFT order parameters describing the nonlinear dynamics.

These claims are tightly validated with numerical simulations, confirming asymptotic error scaling and the correctness of the theoretical predictions for both diagonal and off-diagonal elements.

Figure 1: Time-lagged cross-covariances (scaled by $\sqrt{N}$) for off-diagonal pairs, with simulation (solid) and ansatz (dashed), showing improved fidelity as $N$ increases.

Theoretical Framework and Cavity Derivations

Two complementary derivations are articulated:

1. Linear-Residual Decomposition

Each unit's activity is decomposed as the sum of its linear response to the local field and a nonlinear residual. The central claim is that, for large $N$, the cross-covariance of the residuals at distinct sites is suppressed below $O(1/\sqrt{N})$—in fact, it scales as $O(1/N)$ or faster—indicating that the residual can be treated as effectively independent noise.

Expressed formally, the covariance matrix of the nonlinear RNN is

$N$0

with $N$1 nearly diagonal for large networks; the diagonal elements are controlled by DMFT, while off-diagonal components vanish rapidly. This directly yields the linear equivalence.

2. Self-Consistent Matrix Equation via Two-Site Cavity Method

Here, the joint statistics of pairs of units are characterized using a two-site cavity approach which isolates a pair from the network and analyzes both the Gaussian and non-Gaussian contributions to their joint field statistics. The naive assumption of joint Gaussianity fails to capture key corrections. Proper treatment recovers a self-consistent matrix equation whose solution is exactly the linear-analog covariance, with the correct leading-order and subleading corrections incorporated.

These methods collectively demonstrate that the nonlinearities, while dramatically reshaping microstate trajectories, organize at the level of pairwise (second-order) statistics into a form indistinguishable from a noise-driven linear system, once appropriate self-consistent order parameters are supplied.

Numerical Verification

The ansatz precision is exhaustively benchmarked in simulation, demonstrating:

• Off-diagonal root mean square (RMS) error between theory and simulation decays as $N$2.
• Relative error, after normalizing by the magnitude of off-diagonal elements, decays as $N$3.

This scaling holds robustly through a broad range of $N$4 and sampling ratios, even deep in the chaotic regime, as shown below.

Figure 2: Off-diagonal RMS error and its scaling with $N$5 (panel a/b), and off-diagonal RMS for the residual showing $N$6 suppression (panel c).

The residual's off-diagonal suppression is in fact empirically tighter—around $N$7—than the $N$8 bound derived, reflecting further cancellation due to symmetry in the nonlinearity. However, the overall covariance error is still governed by the $N$9 scaling, dominated by the diagonal contributions.

Implications for Random Matrix Theory and Population Geometry

This equivalence yields numerous theoretical and practical advances:

• Full Spectrum Access: The eigenvalue spectrum of the population covariance, previously only accessible for linear systems, can now be analyzed for generic nonlinear RNNs using established random matrix results (e.g., circular law, Fuss–Catalan, free probability), but with effective parameters from nonlinear DMFT substituted in.
• Dimensionality & Participation: Statistical measures such as effective dimension and participation ratio, central to understanding neural population geometry, are directly computable.
• Analytical Reduction: High-dimensional nonlinear questions collapse onto linear random matrix ensembles, making a wide range of formerly intractable analyses accessible.

  Figure 3: Diagonal autocovariances $O(1/\sqrt{N})$0 compared with DMFT (black), confirming single-site self-averaging.

Relation to Broader High-Dimensional Phenomena

This work generalizes and strengthens recent universality results in nonlinear random matrix theory, which have shown various nonlinear high-dimensional objects (kernel matrices, Gram matrices in random features, etc.) are statistically equivalent to linear-noise analogs in the limit of large dimension and i.i.d. disorder. However, those works were fundamentally feedforward; the current work tackles genuine recurrent disorder and its associated statistical dependencies via the cavity method, establishing equivalence at the finer-grained level of matrix elements (not just spectral statistics or summary measures).

Limitations and Assumptions

The equivalence holds strictly for networks with i.i.d. random couplings, in regimes absent of macroscopic order (e.g., symmetry-breaking, structured patterns), and for realizations typical of the ensemble. Networks with correlated disorder, low-rank structure, or non-asymptotic scaling will exhibit corrections or breakdown of this mapping.

Future Directions

This analytic bridge opens potential for:

• Rigorous analysis of learning-induced and structure-induced deviations from randomness.
• Extension to partially symmetric connectivity or structured noise.
• Investigation of universality classes in more complex correlated or hierarchical RNNs.
• Diagrammatic, path-integral, or non-perturbative generalizations to capture non-self-averaging observables beyond the covariance.

Conclusion

This paper rigorously establishes, via two independent analytic routes and numerical corroboration, that large nonlinear RNNs exhibit an exact linear equivalence for their population covariance at the matrix-element level, with nonlinear dynamics entering solely through scalar DMFT order parameters. This greatly broadens the class of tractable questions regarding the high-dimensional geometry and statistics of activity in nonlinear recurrent systems, and clarifies the mechanism by which complex, collective structure emerges from simple, random coupling rules—reducing the problem to one of effective linear statistics plus noise, despite the underlying nonlinear chaos.