Neural Tangent Kernels and Fisher Information Matrices for Simple ReLU Networks with Random Hidden Weights

Abstract: Fisher information matrices and neural tangent kernels (NTK) for 2-layer ReLU networks with random hidden weight are argued. We discuss the relation between both notions as a linear transformation and show that spectral decomposition of NTK with concrete forms of eigenfunctions with major eigenvalues. We also obtain an approximation formula of the functions presented by the 2-layer neural networks.

• The paper establishes an explicit spectral decomposition of the NTK and Fisher information matrix in two-layer, bias-free random ReLU networks.
• It demonstrates that the network’s response is dominated by specific radial and coordinate eigenfunctions with clearly quantified eigenvalues.
• The analysis reveals practical implications for function approximation, learning dynamics, and sample complexity in the random-feature regime.

Neural Tangent Kernel and Fisher Information for Simple ReLU Networks with Random Hidden Weights

Problem Setting and Objectives

The paper analyzes the spectral properties and interrelations of the Fisher information matrix and neural tangent kernel (NTK) for a class of two-layer, bias-free ReLU neural networks where hidden weights are randomly initialized and fixed (i.e., random features regime). The input $x$ is drawn from a $d$-dimensional standard Gaussian, hidden weights $W \in \mathbb{R}^{d \times m}$ are sampled from $\mathcal{N}(0, 1/m)$, and the task reduces to estimating output-layer weights $v \in \mathbb{R}^m$ with an $\ell_2$ constraint. The NTK is characterized in the limit $m \rightarrow \infty$. Previous literature, e.g., Takeishi et al. (Takeishi et al., 2021, Ho et al., 23 May 2025), revealed strong bias in the spectral distribution of the Fisher information matrix in this setting; this paper establishes an analogous and explicit spectral decomposition for the NTK, detailing eigenfunctions and eigenvalues with concrete formulae.

Fisher Information Matrix and NTK: Formal Definitions and Spectral Structure

Given the model $y = \phi(xW)v^T + \epsilon$ with $\epsilon \sim \mathcal{N}(0,1)$, the Fisher information matrix $J$ for output weights $v$ is: $$J_{ij} = \mathbb{E}_{x}[\phi(xW_{*i}) \phi(xW_{*j})] = \mathbb{E}[X_i X_j],$$ where $X = \phi(xW)$. $J$ is independent of $v$ and admits an approximate spectral decomposition involving highly biased eigenvalues, with eigenvectors constructed from columns of $W$.

The NTK is defined in the infinite-width limit as: $k(x,y) = \mathbb{E}_Z[\phi(xZ^T)\phi(yZ^T)],$ where $Z$ is standard Gaussian, yielding the operator $K$ on the L2 function space. The paper relates $J$ and $K$ as linear transformations between output weight space and function space.

Main Results: Explicit Spectral Decomposition of NTK

The NTK is shown to admit the following approximate expansion: $$k(x, y) = \frac{2d+1}{4\pi} F_0(x)F_0(y) + \frac{1}{4}\sum_{l=1}^d F_l(x)F_l(y) + \frac{1}{2\pi(d+2)} \Biggl( \sum_\gamma F_\gamma(x)F_\gamma(y) + \sum_{\alpha<\beta} F_{\alpha\beta}(x)F_{\alpha\beta}(y) \Biggr) + r(x,y),$$ where:

• $F_0(x) = |x|/\sqrt{d}$ (radial function),
• $F_l(x) = x_l$ (coordinate functions),
• $F_{\alpha\beta}(x) = x_\alpha x_\beta / |x|$ (cross terms),
• $F'_\gamma(x) = (x_\gamma^2 - x_1^2)/|x|$ (variance-type terms, orthogonalized).

These functions form an orthonormal system under $L^2(p_X)$, and are shown by explicit calculation to be eigenfunctions of the NTK operator. The non-leading residual $r(x, y)$ is absolutely convergent and contributes only a small fraction ($<2.6\%$ of the total spectral norm for large $d$) to the kernel trace.

Numerical significance: The leading eigenvalue $\mu_0 \approx (2d+1)/4\pi$ corresponds to $F_0$, with coordinate eigenfunctions $F_l$ at eigenvalue $1/4$, and the cross/variance group $F_{\alpha\beta}, F'_\gamma$ at eigenvalue $\mu_2 \approx 1/(2\pi(d+2))$. The strong concentration of spectral mass in these terms demonstrates that the space of functions representable by the random ReLU network is heavily dominated by a few simple modes—radial and coordinate directions.

Relationship Between NTK and Fisher Information

The NTK and Fisher information matrix $J$ share analogous spectral decompositions, interconnected by isometric mappings. The eigenvectors of $J$ in output weight space correspond to eigenfunctions of NTK in function space. For the random hidden weights regime, the NTK essentially projects arbitrary functions onto the span of these dominant modes, and the Fisher metric becomes equivalent to an L2 metric on this subspace. This connection yields explicit formulae for Kullback-Leibler divergence and learning dynamics in terms of the spectral structure.

Function Approximation and Learning Implications

The derived spectral structure permits an explicit approximation for any function presented by the network: $$f_v(x) \approx f_{\tilde{\theta}^{(D)}}(x) = \sqrt{\mu_0} \theta_0 F_0(x) + \frac{1}{2} \sum_l \theta_l F_l(x) + \sqrt{\mu_2} \left( \sum_\gamma \theta_\gamma F_\gamma(x) + \sum_{\alpha<\beta} \theta_{\alpha\beta} F_{\alpha\beta}(x) \right),$$ where $\tilde{\theta}$ comprises the spectral mode coefficients, and the approximation error is tightly bounded in terms of the neglected spectral tail.

Bold implication: The paper makes the highly explicit claim that the simple random ReLU network is inherently biased towards simpler function forms (radial and coordinate modes), and the sample complexity for reliably estimating weights in each mode scales inversely with the corresponding eigenvalue. Sophisticated, high-frequency modes are suppressed unless the hidden layer size is substantially larger than $d^2$.

Theoretical and Practical Impact, Future Directions

The findings reinforce classical machine learning principles regarding inductive bias: random features with ReLU activations (in the infinite-width limit) induce a strong bias towards low-complexity, simple functional forms. This provides a rigorous foundation for understanding the generalization properties and learning dynamics in such models, connecting back to information-theoretic analyses (e.g., natural gradient, KL divergence, MDL) [TIT23, (Takeishi et al., 2024)].

Practically, the explicit spectral decomposition of NTK and Fisher information matrix enables efficient model selection, error estimation, and theoretical analysis of learning algorithms which operate under random-feature regimes. The explicit formulas offer the prospect of further analytic studies in model capacity, generalization bounds, and regime transitions (e.g., double descent phenomena [Belkin et al.]).

As future work, extending these results to multi-layer random networks, non-Gaussian inputs, and correlated weight matrices may advance the characterization of inductive bias in deep learning architectures. The spectral approach detailed herein may also facilitate the design of random-feature models with tailored inductive properties.

Conclusion

This paper rigorously characterizes the spectral decomposition of the NTK and Fisher information matrix for a simple random ReLU network, deriving explicit eigenfunctions, quantifying the strongly biased eigenvalue distribution, and linking these to analytic expressions for function approximation and learning dynamics. The results elucidate the inherent bias in random-feature neural networks, quantifying both the theoretical and practical consequences for model complexity and generalization.