More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize

Abstract: Of theories for why large-scale machine learning models generalize despite being vastly overparameterized, which of their assumptions are needed to capture the qualitative phenomena of generalization in the real world? On one hand, we find that most theoretical analyses fall short of capturing these qualitative phenomena even for kernel regression, when applied to kernels derived from large-scale neural networks (e.g., ResNet-50) and real data (e.g., CIFAR-100). On the other hand, we find that the classical GCV estimator (Craven and Wahba, 1978) accurately predicts generalization risk even in such overparameterized settings. To bolster this empirical finding, we prove that the GCV estimator converges to the generalization risk whenever a local random matrix law holds. Finally, we apply this random matrix theory lens to explain why pretrained representations generalize better as well as what factors govern scaling laws for kernel regression. Our findings suggest that random matrix theory, rather than just being a toy model, may be central to understanding the properties of neural representations in practice.

• The paper challenges classical generalization theories by showing that norm-based measures fail as model complexity increases.
• It demonstrates that the generalized cross-validation estimator reliably predicts risk across diverse architectures and datasets.
• The study reveals that aligning with a local Marchenko-Pastur law highlights random matrix effects as central to neural network generalization.

Summary of "More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize"

The paper "More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize" investigates the applicability of random matrix theory in understanding the generalization properties of neural networks, especially in overparameterized settings. This work challenges existing theories by proposing that the generalized cross-validation (GCV) estimator effectively predicts generalization risks, contrary to traditional norm- and spectrum-based analyses that fall short in such contexts.

Key Findings

1. Generalization Challenges with Current Theories: The paper identifies significant challenges with classical theories in explaining the generalization of overparameterized models. The authors spotlight the inadequacy of norm-based measures, which tend to become vacuous as the norms of solutions grow with sample size. They also argue that approaches relying on the convergence of empirical covariance matrices to their population counterparts are flawed due to slow convergence rates in high-dimensional spaces.
2. Effectiveness of the GCV Estimator: The authors demonstrate that the GCV estimator can reliably predict the generalization risk across various architectures and datasets, including situations with significant train-test gaps. This finding is empirically validated with experiments involving realistic neural representations such as eNTKs derived from large neural networks.
3. Theoretical Justification via Random Matrix Theory: The paper provides a theoretical basis for the empirical success of GCV by showing that it aligns well with predictions made under a local version of the Marchenko-Pastur law. This highlights that random matrix effects are not mere artifacts but central to understanding generalization in modern machine learning models.
4. Implications for Pretraining and Scaling Laws: The research extends its random matrix theory perspective to provide insights into why pretrained models often generalize better than randomly initialized ones. It finds that, although pretrained models have slower eigendecays, they exhibit superior alignment properties with ground truths, leading to better generalization. Moreover, the study proposes new methods to estimate the power law scaling rates necessary to understand the empirical scaling laws observed in large-scale neural models.

Implications and Future Directions

The findings have significant implications for understanding neural networks, particularly in high dimensions. By putting random matrix theory at the forefront, the authors suggest moving beyond classical approaches and focusing on understanding spectral properties and alignments in the model representations. The insights provided could pave the way for more efficient pretraining strategies and more accurate scaling laws predictions, crucial for developing and deploying large-scale neural networks.

The work encourages further exploration into:

• Applying the random matrix theory framework to other learning settings (e.g., logistic regression, covariate shift models).
• Investigating the evolution of neural tangent kernels during training to gain insights into feature learning processes.
• Enhancing theoretical conditions under which random matrix laws apply, especially in non-asymptotic regimes relevant to practical machine learning tasks.

Overall, this study offers a robust challenge to conventional approaches dealing with generalization in machine learning, advocating for a paradigm shift towards random matrix models that can more accurately characterize and predict the complex behaviors of deep neural networks.