Deep Learning is Not So Mysterious or Different

Abstract: Deep neural networks are often seen as different from other model classes by defying conventional notions of generalization. Popular examples of anomalous generalization behaviour include benign overfitting, double descent, and the success of overparametrization. We argue that these phenomena are not distinct to neural networks, or particularly mysterious. Moreover, this generalization behaviour can be intuitively understood, and rigorously characterized, using long-standing generalization frameworks such as PAC-Bayes and countable hypothesis bounds. We present soft inductive biases as a key unifying principle in explaining these phenomena: rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data. This principle can be encoded in many model classes, and thus deep learning is not as mysterious or different from other model classes as it might seem. However, we also highlight how deep learning is relatively distinct in other ways, such as its ability for representation learning, phenomena such as mode connectivity, and its relative universality.

• The paper shows that generalization phenomena like benign overfitting and double descent occur in both deep networks and simpler linear models.
• It employs soft inductive biases and order-dependent regularization to explain robust generalization without relying on model-specific architecture constraints.
• The study distinguishes deep networks through unique aspects such as representation learning, mode connectivity, and in-context learning.

Authoritative Essay on "Deep Learning is Not So Mysterious or Different" (2503.02113)

Introduction

"Deep Learning is Not So Mysterious or Different" systematically deconstructs the prevailing narrative that deep neural networks exhibit unique, enigmatic generalization behavior unlike other model classes. The paper rigorously examines phenomena such as benign overfitting, double descent, and the effectiveness of overparametrization, positing that these attributes are not exclusive to deep learning and are readily reproduced and understood via simple linear models and established theoretical frameworks. The central thesis is that soft inductive biases, rather than hard architectural constraints, provide a coherent blueprint for achieving robust generalization across all model classes. The discourse is further expanded to delineate genuinely distinctive characteristics of deep networks, such as representation learning, mode connectivity, and universality in in-context learning.

Generalization Phenomena: Bridging Deep Learning and Simple Models

The paper presents compelling evidence that key generalization phenomena attributed to neural networks are neither exclusive nor mysterious. Benign overfitting and double descent are illustrated with high-order polynomials and linear models, emphasizing that the phenomena can be replicated outside deep learning. A $150$th order polynomial with order-dependent regularization demonstrates benign overfitting, fitting both structured and unstructured data while preserving generalization; Gaussian processes are shown to mimic CNN behavior on CIFAR-10, including the capacity to fit noisy labels with maintained performance.

Figure 1: Generalization phenomena associated with deep learning can be reproduced with simple linear models and understood.

Additionally, double descent is observed in both ResNet architectures and linear random feature models—highlighting the phenomenon's independence from neural network-specific mechanisms.

Formal Generalization Frameworks: PAC-Bayes and Compressibility

The paper asserts that classical frameworks such as VC dimension and Rademacher complexity are insufficient to explain these phenomena due to their reliance on hypothesis space cardinality. By contrast, PAC-Bayes and countable-hypothesis bounds encapsulate the effect of compressibility and simplicity biases, offering non-vacuous generalization guarantees for models with millions or billions of parameters. The empirical risk and compressibility—quantified via Kolmogorov complexity—upper bound generalization, providing a procedural recipe for learning: maximize hypothesis flexibility while biasing towards compressible solutions.

Figure 2: Generalization phenomena can be formally characterized by generalization bounds anchored in empirical risk and compressibility.

This approach aligns with Solomonoff induction, which assigns exponential preference to simpler programs, ensuring that, even in maximally flexible hypothesis spaces, compressible (hence simple) solutions generalize.

Soft Inductive Biases: Beyond Restriction

A unifying theme is the concept of soft inductive biases, which prefer rather than restrict certain solutions, enabling flexible hypothesis spaces without deleterious overfitting. Multiple mechanisms implement soft biases: order-dependent regularization, architectural design (e.g., residual pathway priors), and implicit Bayesian priors. The paper presents order-dependent regularization in polynomials as a canonical example, showing that the model adaptively selects simpler explanations in low-data regimes and flexibly fits complex data as needed; regularized high-order polynomials perform as well or better than constrained models across varying data complexities and sizes.

Figure 3: Soft inductive biases enlarge the hypothesis space with preferences for particular solutions over others, enabling flexible learning without overfitting.

Figure 4: Achieving good generalization with soft inductive biases involves steering optimization toward preferred solutions within a rich hypothesis space.

Figure 5: Flexibility combined with simplicity bias enables appropriate model adaptation across diverse problem complexities and dataset sizes.

Overparametrization and Double Descent: Flexibility Amplifies Compression Bias

The paper provides a nuanced perspective on overparametrization, repudiating parameter counting as a valid complexity metric. Larger models not only increase flexibility but also enhance compressibility: flat minima occupy exponentially greater volumes in the parameter space, and are more discoverable via optimization, fostering a simplicity bias. Empirical studies highlighted show that wide models manifest lower effective dimensionality and stronger compression (Kolmogorov complexity), ultimately improving generalization, even when training achieves zero loss.

Figure 6: Increasing parameters promotes generalization by increasing the volume of flat, compressible solutions.

Double descent is contextualized as a transition from classical bias-variance regimes to interpolation regimes, wherein further parameter increases favor simpler (low-norm) solutions among interpolants, improving generalization. The phenomenon is not unique to neural networks and has historical precedents in statistical models and random matrix theory.

Genuine Distinctions of Deep Learning: Representation, Universality, and Mode Connectivity

Although generalization phenomena are not mysterious, deep networks remain distinctive in several respects:

• Representation Learning: Neural networks adaptively learn basis functions, dynamically capturing relevant features for high-dimensional signals, surpassing fixed-kernel or basis models. This ability underpins successful extrapolation and nuanced similarity metrics tailored to complex tasks.
• Universality and In-Context Learning: Deep networks, especially transformers, exhibit a compression bias closely matching the real-world data distributions, resulting in universal applicability across modalities and tasks. In-context learning is robust, with pre-trained models achieving competitive zero-shot performance on diverse domains, explained via flexible mixture-of-experts analogies.
• Mode Connectivity: Loss landscapes in deep networks exhibit connected modes, allowing traversal between independently trained solutions along simple curves with negligible loss increase. This property contradicts a conventional view of isolated local minima, facilitating model merging (e.g., SWA) and providing new avenues for understanding generalization.

Figure 7: Modes in deep neural network loss landscapes are connected along curves, not isolated, a property relatively distinct to deep networks.

Implications and Speculative Outlook

The findings challenge foundational assumptions in machine learning theory, advocating a move away from rigid restriction biases and parameter counting. Theoretical frameworks embracing compressibility and soft inductive biases offer scalable, robust guarantees for both neural and non-neural models. Practically, this supports development of model-agnostic learners and motivates research in adaptive priors and compressibility-aware architectures. Future directions include: deeper empirical evaluation of generalization bounds, exploring measures beyond Kolmogorov complexity, and optimizing soft bias implementations for computational efficiency.

Furthermore, distinctive properties such as representation learning and mode connectivity suggest continued progress toward universality and adaptive intelligence in AI, modulated by compressibility rather than explicit architecture constraints.

Conclusion

The paper "Deep Learning is Not So Mysterious or Different" articulates an integrative framework that demystifies generalization behavior observed in deep learning, grounding it in general principles applicable to a broad spectrum of models. Overparametrization, benign overfitting, and double descent are traced to soft inductive biases and compressibility, rather than inherent neural network properties. The theoretical and empirical analyses reinforce the universality of these concepts, while distinguishing neural networks in representation learning, mode connectivity, and in-context universality. This synthesis reframes deep learning as remarkable not for its mysterious generalization, but for its practical expressivity and adaptability, signaling a paradigm shift in both theoretical understanding and model design.