Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learning

Abstract: We provide an overview of high dimensional dynamical systems driven by random matrices, focusing on applications to simple models of learning and generalization in machine learning theory. Using both cavity method arguments and path integrals, we review how the behavior of a coupled infinite dimensional system can be characterized as a stochastic process for each single site of the system. We provide a pedagogical treatment of dynamical mean field theory (DMFT), a framework that can be flexibly applied to these settings. The DMFT single site stochastic process is fully characterized by a set of (two-time) correlation and response functions. For linear time-invariant systems, we illustrate connections between random matrix resolvents and the DMFT response. We demonstrate applications of these ideas to machine learning models such as gradient flow, stochastic gradient descent on random feature models and deep linear networks in the feature learning regime trained on random data. We demonstrate how bias and variance decompositions (analysis of ensembling/bagging etc) can be computed by averaging over subsets of the DMFT noise variables. From our formalism we also investigate how linear systems driven with random non-Hermitian matrices (such as random feature models) can exhibit non-monotonic loss curves with training time, while Hermitian matrices with the matching spectra do not, highlighting a different mechanism for non-monotonicity than small eigenvalues causing instability to label noise. Lastly, we provide asymptotic descriptions of the training and test loss dynamics for randomly initialized deep linear neural networks trained in the feature learning regime with high-dimensional random data. In this case, the time translation invariance structure is lost and the hidden layer weights are characterized as spiked random matrices.

• The paper demonstrates that DMFT effectively models high-dimensional disordered systems by linking random matrices with machine learning dynamics.
• It employs advanced methods like the cavity method and path integrals to derive asymptotic descriptions of training and test loss in deep linear networks.
• It further analyzes bias-variance trade-offs and ensemble effects, offering insights for optimizing neural network training protocols.

Disordered Dynamics in High Dimensions

This essay summarizes the paper "Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learning" (2601.01010). The paper explores high-dimensional dynamical systems characterized by randomness, focusing on applications in machine learning, particularly learning and generalization models. It leverages techniques from the cavity method and path integrals to analyze how infinite-dimensional coupled systems behave as stochastic processes.

Dynamical Mean Field Theory (DMFT)

The paper provides a detailed examination of dynamical mean field theory (DMFT), which is crucial for understanding high-dimensional, disordered systems. DMFT characterizes single-site stochastic processes using a set of correlation and response functions. These functions provide insight into how perturbations are remembered within the system. Linear time-invariant systems: DMFT elucidates connections between random matrix resolvents and response functions, demonstrating applications in simplified learning models such as gradient flow and stochastic gradient descent.

Applications in Machine Learning

Random Feature Models and Deep Networks: DMFT is applied to investigate machine learning models, including random feature models and deep linear networks trained in feature-learning regimes. These models utilize random matrices to simulate training dynamics on random data, revealing complex behavior like non-monotonic loss curves influenced by non-Hermitian matrices. The paper offers asymptotic descriptions of training and test loss dynamics in deep linear networks initialized randomly, illustrating how feature-learning structure deviates from lazy training.

Figure 1: Cavity derivation of the marginal dynamics for a single site of the system as $N \to \infty$. Adding a new site to the system comes with reciprocal couplings.

Advanced Theoretical Insights

The paper discusses several complex theoretical constructs, such as the bias-variance decomposition and ensembling techniques in DMFT. Bias and variance computations: Through averaging over DMFT noise variables, the paper calculates bias and variance components, facilitating a deeper understanding of ensemble learning and bagging effects.

Figure 2: The DMFT response function for the random Wigner matrix encodes the semicircle eigenvalue law; pictorial representation of spectral distribution.

Implications and Future Research Directions

The findings have profound implications for both practical applications in AI and theoretical advancements. Practically, improved comprehension of feature-learning mechanics aids in optimizing training protocols for neural networks. Theoretically, the paper challenges conventional understanding by showing that DMFT can model non-linear, non-Hermitian dynamics effectively, suggesting avenues for future exploration.

Figure 3: Visualization and detailed illustration of linear regression problem dynamics.

Conclusion

The paper "Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learning" comprehensively examines how randomness influences high-dimensional dynamical systems. By applying dynamical mean field theory, it provides a framework for analyzing how these systems operate, particularly within the context of machine learning models. This paper marks a significant step in modeling complex, non-linear dynamics, offering promising directions for future theoretical and empirical research in AI.