Neural Tangent Hierarchy: NTK-ECRN Analysis

• Neural Tangent Hierarchy (NTH) is a framework that uses Fourier feature embeddings, layerwise scaling, and stochastic depth to precisely control the NTK spectrum in deep residual networks.
• The design enables analytic tracking of eigenvalue evolution and bounds NTK drift, ensuring stable optimization and improved generalization during gradient-based training.
• Empirical evaluations demonstrate that NTK-ECRN outperforms traditional models in regression, classification, and benchmark tasks by achieving lower error rates and stable spectral behavior.

The NTK-Eigenvalue-Controlled Residual Network (NTK-ECRN) is a residual network architecture engineered to admit direct control and rigorous analysis of its Neural Tangent Kernel (NTK) spectrum, which enables explicit manipulation of generalization and optimization dynamics via spectral methods. The NTK-ECRN amalgamates Fourier feature input embeddings, residual connections with layerwise scaling, and stochastic depth to regulate the evolution of the NTK, and—critically—of its eigenvalue distribution during gradient-based training. The following sections describe its formal structure, spectral and theoretical properties, eigenvalue behavior, connections to established NTK/ResNet results, key empirical findings, and broader implications within neural tangent kernel theory and deep learning (Mysore et al., 9 Dec 2025, Li et al., 2020, Belfer et al., 2021, Littwin et al., 2020).

1. Formal Structure of NTK-ECRN

The NTK-ECRN is an $L$-layer residual network parameterized to control its NTK spectrum through architectural components and explicit scaling schemes:

• Fourier Feature Embedding: Each input $x\in\mathbb{R}^d$ is mapped via fixed (or learnable) frequency matrix $B\in\mathbb{R}^{d_f\times d}$ to a higher-dimensional vector

$$\phi(x) = [{\sin(2\pi Bx)},\;{\cos(2\pi Bx})] \in \mathbb{R}^{2d_f}$$

to support high-frequency eigenmodes.

• Residual Blocks with Layerwise Scaling: For $l=1,\ldots,L$, each block computes

$$h^{(l)} = h^{(l-1)} + \alpha_l\,\sigma\big(W^l h^{(l-1)} + b^l\big)$$

where $\sigma$ is a smooth nonlinearity (e.g., $\tanh$, GELU), $\alpha_l>0$ is a controllable scaling factor, $W^l\in\mathbb{R}^{n\times n}$, $b^l\in\mathbb{R}^n$.

• Stochastic Depth: Optionally, block $l$ is dropped with probability $p_l$, introducing stochastic regularization:

$$h^{(l)} = h^{(l-1)} + m_l\,\alpha_l\,\sigma\big(W^l h^{(l-1)} + b^l\big),\quad m_l \sim \text{Bernoulli}(1-p_l).$$

• Initialization: Standard NTK initialization is used, with

$$W^l_{ij} \sim \mathcal{N}(0,1/n), ~~ b^l_i \sim \mathcal{N}(0,1/n)$$

to ensure convergence to a deterministic NTK in the $n\rightarrow\infty$ limit.

• Output Layer: The final output is $\hat y = W^{L+1} h^{(L)} + b^{(L+1)}$.

These choices directly prescribe spectral properties of the associated NTK (Mysore et al., 9 Dec 2025).

2. NTK Dynamics and Eigenvalue Evolution

At training time $t$, the sample-wise NTK is

$$K_t(x,x') = \nabla_\theta f_\theta^{(t)}(x)\cdot\nabla_\theta f_\theta^{(t)}(x') = \sum_{l=1}^{L} \frac{\partial f_\theta^{(t)}(x)}{\partial W^l} \frac{\partial f_\theta^{(t)}(x')}{\partial W^l}^\top.$$

Let $\Theta_t$ denote the $n\times n$ Gram matrix over $n$ data points.

• Frobenius Norm Bound: The evolution of $\Theta_t$ is tightly controlled,

$$\|\Theta_{t+1}-\Theta_0\|_F \le \|\Theta_t-\Theta_0\|_F + \alpha_l^2 \|\sigma'\|_\infty^2,$$

which globally yields

$$\|\Theta_t-\Theta_0\|_F \le t\,\max_{1\leq l\leq L} \big(\alpha_l^2 \|\sigma'\|_\infty^2\big).$$

• Eigenvalue Evolution: For the eigenvalues $\lambda_i(t)$ of $\Theta_t$,

$$|\lambda_i(\Theta_{t+1}) - \lambda_i(\Theta_t)| \le \|B\|_2,$$

with $B$ the rank-one Gram update per layer, thereby bounding the per-step fluctuation of both dominant and minor eigenvalues.

• Dominant Eigenvalue Recurrence:

$$\lambda_{\max}(\Theta^{(l+1)}_t) \le \lambda_{\max}(\Theta^{(l)}_t) + \alpha_l^2 \Vert J_t^{(l)} \Vert_2^2,$$

with $$J^{(l)}_t(x) = \partial(\sigma(W^l h^{(l-1)}+b^l))/\partial\theta$$.

These results enable analytic tracking of NTK drift and eigenvalue trajectories throughout optimization (Mysore et al., 9 Dec 2025).

3. Spectral Properties, Generalization, and Conditioning

The NTK spectrum governs both function-space expressivity and optimization stability:

• Generalization Bound: In the NTK eigenbasis, the generalization error satisfies

$$\mathcal{E}_{\text{gen}} \leq \sum_{i=1}^{n}\frac{(f_i - y_i)^2}{\lambda_i} + \varepsilon,$$

where large eigenvalues $\lambda_i$ facilitate improved generalization for corresponding eigendirections.

• Optimization Stability: The condition number $\kappa(\Theta_t) = \lambda_1(t)/\lambda_n(t)$ is moderated by judicious $\{\alpha_l\}$ and $p_l$ choices, ensuring absence of "edge-of-stability" phenomena, i.e., abrupt $\lambda_1$ spikes.
• Role of Components:
  • Larger $\alpha_l$ amplify high-frequency eigenmodes but must be capped to avoid spectrum blow-up.
  • Fourier feature embeddings enhance the initial kernel support for high-frequency components, flattening initial $\{\lambda_i(0)\}$ decay.

By tuning these parameters, NTK-ECRN achieves spectral sculpting across training and model scaling regimes (Mysore et al., 9 Dec 2025).

4. Comparison to Residual Network NTK Theory

The NTK-ECRN extends and operationalizes rigorous results obtained for ResNet NTK and related random kernel architectures:

• Polynomial Width Scalings: Standard residual networks with analytic, Lipschitz activations and skip connections require only $m = O(n^3 L^2 \log^2(1/\epsilon))$ width (for training set size $n$, depth $L$, and error floor $\epsilon$), removing the exponential-in-$L$ scaling barrier for generalization and kernel stability found in plain feedforward networks (Li et al., 2020).
• Spectrum Decay and Harmonization: In infinite width, the NTK eigenfunctions (for inputs on the sphere) of residual architectures are spherical harmonics, and eigenvalues decay polynomially as $k^{-d}$ for frequency $k$ and input dimension $d$, matching FC-NTK and Laplace kernel RKHSs (Belfer et al., 2021).
• Spectral Control via Scaling: Layerwise scalings $\alpha_l$ determine whether the spectrum is stable (flat, nondegenerate for $\alpha_l=O(1/L)$ or $\alpha=L^{-\gamma}$, $\gamma>0.5$) or "sharpens" into spike-like pathology (for fixed $\alpha_l$ as $L\to\infty$). Stable spectra avoid degeneracy and parity bias, maintaining depth-robust accuracy (Belfer et al., 2021, Littwin et al., 2020).

The NTK-ECRN generalizes these insights by further leveraging Fourier feature pre-conditioning and stochastic depth regularization as explicit mechanisms for spectrum tuning (Mysore et al., 9 Dec 2025).

5. Finite-Width Corrections and Practical Design Guidelines

Finite width induces $O(1/n)$ corrections to both the Gramian and spectrum. More precisely, eigenvalues satisfy

$$\lambda_i(n) = \lambda_i(\infty) + \frac1n\,\delta\lambda_i + O(n^{-2})$$

and the condition number degrades only by $O(1/n)$—provided

$\frac{5m + \sum_{l}\alpha_l}{n} \lesssim 1.$

For standard scaling ($\alpha_l=1/L$), this yields spectrum preservation even for deep networks (Littwin et al., 2020). With improper scaling (e.g., large $\alpha_l$ or $L/n \gtrsim 1$), the spectrum can sharply "explode" or "collapse," degrading trainability and expressivity.

Stochastic depth further limits finite-width fluctuations by regularizing the kernel drift and increasing analytic tractability (Mysore et al., 9 Dec 2025).

6. Empirical Results

Empirical studies confirm the NTK-ECRN's theoretical properties:

• On synthetic regression ($d=20$, 10 Fourier modes), the NTK-ECRN achieves the lowest MSE ($0.045\pm0.004$) and highest $R^2$ ($0.92\pm0.01$) among MLP, ResNet-18, and standard NTK baselines.
• On synthetic classification (5 Gaussian classes), NTK-ECRN attains $93.8\pm0.7\%$ accuracy and $0.312\pm0.010$ CE loss, outperforming all baselines.
• On tabular UCI benchmarks, NTK-ECRN yields $2$–$5$ point gains in $R^2$ (Boston Housing) or accuracy (Iris, Wine) over competitors.
• On CIFAR-10 subset (5,000 images), NTK-ECRN achieves $81.9\%$ accuracy and $0.648$ CE loss, exceeding ResNet-18, MLP, and standard NTK models.
• Spectral analysis during training shows the maximal eigenvalue $\lambda_1(t)$ evolves smoothly (no spiking), and $\|\Theta_t-\Theta_0\|_F$ grows linearly with $t$ as predicted.

These results confirm practical NTK spectrum control translates to improved stability and generalization in diverse settings (Mysore et al., 9 Dec 2025).

7. Broader Implications and Perspectives

NTK-ECRN establishes a framework for bridging infinite-width NTK theory with practical (finite-width) deep learning models by:

• Embedding Fourier features for initialization spectrum shaping
• Applying explicit layerwise residual scaling for NTK drift bounding
• Using stochastic depth to enhance regularization and enable analytic kernel dynamics

Potential extensions include adaptive scheduling of $\{\alpha_l\}$ informed by NTK eigenvalue monitoring and integration with batch normalization. A key limitation is the persistence of finite-width fluctuations, with error terms $\varepsilon$ increasing as model width shrinks. Tightening non-asymptotic bounds for finite-width regimes remains an open avenue (Mysore et al., 9 Dec 2025).

By enabling analytic and empirical control of spectral evolution, NTK-ECRN provides a principled paradigm for designing deep residual architectures resilient to depth, with tunable generalization and optimization properties throughout training and scaling regimes.