Lecture notes: From Gaussian processes to feature learning

Abstract: These lecture notes develop the theory of learning in deep and recurrent neuronal networks from the point of view of Bayesian inference. The aim is to enable the reader to understand typical computations found in the literature in this field. Initial chapters develop the theoretical tools, such as probabilities, moment and cumulant-generating functions, and some notions of large deviation theory, as far as they are needed to understand collective network behavior with large numbers of parameters. The main part of the notes derives the theory of Bayesian inference for deep and recurrent networks, starting with the neural network Gaussian process (lazy-learning) limit, which is subsequently extended to study feature learning from the point of view of adaptive kernels. The notes also expose the link between the adaptive kernel approach and approaches of kernel rescaling.

• The paper presents a unified framework that bridges Gaussian processes with adaptive feature learning in neural networks through statistical physics and Bayesian analysis.
• It employs large deviation and saddle-point methods to derive analytic expressions for network outputs in both infinite-width and finite data regimes.
• The study provides practical insights into performance prediction and kernel adaptation, offering guidelines for architecture selection and understanding model generalization.

From Gaussian Processes to Feature Learning in Neural Networks

Overview and Objectives

This work provides an extensive, physics-oriented exposition on the theoretical foundations bridging Gaussian processes (GPs) and feature learning in artificial neural networks (ANNs), focusing on both technical and conceptual advances. The notes unify tools from statistical field theory, large deviation principles, and the formal structure of Bayesian learning, connecting classical results on the Gaussian process limit of wide networks to recent developments in feature-learning theory. Special attention is given to the regimes where both width ($N$) and dataset size ($P$) scale in proportion, capturing the emergence of data-dependent representations absent from purely GP-based models.

Foundational Framework and Methodological Approach

The core theoretical setting is the Bayesian paradigm for supervised learning, in which weight priors and data likelihoods induce posterior distributions over network parameters and, consequently, over outputs. This approach is agnostic to specific training algorithms and naturally aligns with equilibrium distributions obtainable via Langevin dynamics, facilitating direct analytic study with field-theoretic techniques.

A central motif is the mapping between machine learning problems and statistical mechanics: the learning process is encoded as an inference over a partition function, with the cost or loss function acting as an energy term and weight configurations forming the microstates. Large-width limits enable the application of saddle-point and large deviation analyses, while treating the network output as the primary object of study—departing from more usual focuses on weight space geometry or function-space capacity.

The notes are organized to first build up moment and cumulant machinery, progress through the construction of joint and conditional distributions, and then ground these in the explicit case of Gaussian processes, which serve as both a mathematical base case and a tractable model for extremely wide networks. This scaffolding supports later chapters tackling the law of large numbers, deep and recurrent architectures, stochastic training dynamics, and, crucially, the mechanisms underlying feature learning.

Gaussian Processes and the Infinite-Width Limit

The formal link between neural networks and Gaussian processes is addressed via the so-called neural network Gaussian process (NNGP) correspondence. In the $N \to \infty$, $P$ fixed limit, the joint distribution of outputs converges to a GP determined by the architecture and initialization, but fundamentally independent of the training labels—an instance of "lazy learning" where internal representations are frozen.

This equivalence is shown explicitly for both shallow and deep feedforward architectures using both heuristic arguments (via central limit theorem) and rigorous field-theoretic derivations. Similar technology is applied to recurrent networks, with appropriate modifications to account for weight sharing across time steps (temporal unfolding).

The Bayesian posterior of the outputs under the NNGP regime yields analytic expressions for the predictive mean and covariance. These are identical to those arising from linear regression with feature maps implied by the network activations at initialization, underscoring the absence of adaptive feature extraction in this regime.

Beyond GPs: Feature Learning and the $P/N$ Scaling

The primary departure from the GP paradigm occurs when $N$ and $P$ scale together with constant ratio $\alpha = P/N$. In this setting, the data-dependent part of the objective function meaningfully perturbs the order parameters controlling the network's internal representations, leading to "feature learning": the adaptation of hidden layer kernels based on correlations between inputs and labels.

Two main theoretical approaches are reviewed and connected:

• Kernel Scaling Theories: The impact of finite $P/N$ is incorporated as a global rescaling of the effective GP kernel, with the scaling parameter (e.g., $\|w\|^2$ in shallow linear cases) determined by a self-consistent equation involving both the prior and the data term [Li et al., 2021]. The solution is obtained via a saddle-point analysis in the large $N$ limit, and connects with phenomena such as neural scaling laws and loss curves.
• Kernel Adaptation Theories: These frameworks allow the GP kernel itself to deform under the influence of data, rather than just being rescaled. The resulting adaptation equations are solved via variational methods or large deviation theory and make manifest the (often subtle) ways in which learned representations diverge from those of random GPs. In particular, the off-diagonal structure and outlier eigenvalues of the learned kernel capture taskspecific information absent from the NNGP [seroussi23_908, Fischer24_10761].

The notes detail how both approaches are mathematically linked, treating the first as a restriction of the latter to scaling solutions, and provide a technical bridge, following [Rubin25], which unifies the two viewpoints.

Dynamical Perspectives and Non-equilibrium Training

A notable section is the formal connection between Bayesian posteriors and training dynamics governed by stochastic Langevin equations. The Fokker-Planck framework is deployed to derive the stationary distribution over weights induced by stochastic gradient descent with noise and weight decay, recovering the analytic Bayesian fixed point. This insight bolsters the legitimacy of the Bayesian/statistical physics lens for analyzing realistic training protocols.

Phase Transitions, Memory, and the Economics of Learning

The field-theoretic treatment enables the identification and quantitative characterization of learning phase transitions as the amount of training data or other control parameters are varied. These transitions—marked by abrupt changes in capacity, generalization, or feature selectivity—are explained in terms of free energy minimization and the classical competition between energy and entropy.

The capacity of wide networks to generalize, as well as their tendency to implement smooth functions (an inductive bias captured in the GP prior), emerge as consequences of these analytic treatments. The framework also sets the stage for studying neural scaling laws, criticality, and the emergence (or failure) of internal representations in the high-capacity regime.

Theoretical and Practical Implications

Theoretical implications include:

• Precise conditions for when neural networks exhibit "lazy" (GP-like) versus "rich" (feature-learning) behavior;
• The identification of sufficient statistics (e.g., kernel order parameters, norms, overlaps) for analyzing and predicting network generalization;
• Clarification of the universality (and its breakdown) of GP behavior as the data-to-width ratio increases.

Practical implications include:

• Improved methods for predicting the performance and uncertainty of Bayesian neural networks without expensive retraining;
• Insights for selecting architectures and hyperparameters based on their kernel adaptation dynamics;
• Guidance for understanding the limits of architectural overparametrization and data efficiency.

The field-theoretic and large deviation approaches provide general techniques potentially extensible to modern architectures, including convolutional, graph-based, and attention-based networks, as currently under active investigation.

Outlook and Future Developments

As neural networks increase in both depth and width, and as applications become more data- and compute-intensive, analytic frameworks such as those described in this work become invaluable for rational model selection, principled uncertainty estimation, and understanding the emergence of intelligence-like behaviors through collective phenomena. Further, the theoretical machinery here points to fertile intersections with random matrix theory, high-dimensional statistics, and modern developments in non-equilibrium statistical mechanics and dynamical systems.

Open directions involve extending these analyses to cover:

• Nonidentically distributed or structured data (distribution shift, transfer learning)
• Explicitly non-Bayesian or non-equilibrium optimization procedures
• Sparsity, pruning, and subnetwork selection
• More expressive or compositional feature learning regimes

Conclusion

These lecture notes synthesize the rigorous theoretical tools required to bridge the gap between the analytically tractable infinite-width, GP regime and the empirically richer world of adaptive feature learning in deep networks. By systematically treating both shallow and deep, feedforward and recurrent architectures with a unified, field-theoretic language, the work lays out both the constraints and flexibilities of current theories, provides a roadmap for future analytic advances, and anchors the ongoing dialogue between statistical physics and modern machine learning (2602.12855).