Grokking Phenomenon in Neural Networks

• Grokking is a two-phase phenomenon where networks first memorize the training data and then suddenly generalize after prolonged training.
• The phenomenon is explained by theories including spectral misalignment, kernel-to-feature transitions, and complexity compression, revealing deep insights into network dynamics.
• Optimization strategies such as spectral equalization and norm regularization can expedite the grokking transition, enhancing model robustness and generalization.

Grokking is a counter-intuitive phenomenon in neural network training, characterized by prolonged overfitting—training loss reaches (near) zero while test loss remains high—followed by an abrupt and dramatic improvement in test performance after many additional optimization steps. This delayed generalization emerges across a variety of architectures (MLPs, transformers, CNNs), tasks (algorithmic, image, parity, group composition), and optimization regimes. Unlike classical overfitting scenarios, where excessive training leads only to degraded generalization, the grokking regime features a clear two-phase training curve: an early memorization plateau and a sudden "grok" event where the model internalizes the underlying rule or structure, enabling robust out-of-sample prediction. Grokking has proven foundational for analyzing deep-network dynamics, algorithmic learning, generalization, and serves as a touchstone for the development of new complexity and robustness metrics, optimizer modifications, and theoretical frameworks.

1. Empirical Phenomenology and Operational Definition

Grokking manifests as a sharp temporal separation between training set memorization and test set generalization. Let $A_{\mathrm{train}}(t)$ and $A_{\mathrm{test}}(t)$ denote the training and test accuracies at time (or iteration) $t$, with loss functions $L_{\mathrm{train}}(t)$ and $L_{\mathrm{test}}(t)$. In the canonical grokking behavior one observes:

• $A_{\mathrm{train}}(t) \to 1$ (or $L_{\mathrm{train}}(t) \to 0$) for $t \gg t_0$,
• $A_{\mathrm{test}}(t) \approx A_{\mathrm{chance}}$ (e.g., $1/\mathrm{\#classes}$) for $t_0 \ll t < t_g$,
• then, at $t \approx t_g$, $A_{\mathrm{test}}(t)$ rapidly increases to $\sim 1$ (or $L_{\mathrm{test}}(t)$ suddenly collapses).

This two-phase dynamic is robust across architectures, including two-layer MLPs, deep MLPs, transformers, CNNs, and graph neural networks, and is observed in both synthetic (modular arithmetic, parity, group operations) and real-world datasets (MNIST, CIFAR-10, CIFAR-100, Imagenette) (Zhou et al., 2024, Fan et al., 2024, Humayun et al., 2024, Zhang et al., 16 May 2025). Grokking plateaus are quantifiable via the time gap $\Delta t = t_{\mathrm{grok}} - t_{\mathrm{mem}}$, where $t_{\mathrm{mem}}$ is the step at which train accuracy saturates.

2. Core Mechanistic Theories

Multiple theoretical frameworks have been advanced to explain grokking. The major lines are:

1. Frequency-Domain Misalignment: Grokking is driven by a two-phase spectral learning process. Neural networks initially fit spurious low-frequency components caused by non-uniform or undersampled training data (spectral aliasing), leading to rapid train loss decrease but poor test performance. Only after prolonged training do the networks fit the true, typically higher-frequency, components required for generalization, aligning spectral modes between train and test sets (Zhou et al., 2024). This explanation is corroborated by Fourier analysis on sinusoidal, Boolean, and image tasks.
2. Generalization–Compression Phases: Networks transition from a high-complexity, memorizing solution to an information-compressed, generalizing one. Metrics such as Linear Mapping Number (LMN) and an algorithmic rate–distortion complexity bound (Kolmogorov-inspired) show a rise during memorization and a sharp fall at grokking, with the test loss linearly tracking model complexity during this compression window (Liu et al., 2023, DeMoss et al., 2024).
3. Kernel-to-Feature-Learning Transitions: Early in training, wide networks behave as kernel predictors (NTK or GP regime), which interpolate training data but require an $\Omega(1)$ fraction of the input space to generalize (impossible with small samples). Grokking coincides with the escape from the kernel/lazy regime into a "rich," feature-learning regime, often mediated by $\ell_\infty$-norm (or margin) bias from weight decay or implicit regularization, enabling efficient generalization even from sparse examples (Mohamadi et al., 2024, Lyu et al., 2023).
4. Glassy Relaxation Analogy: Training is mapped onto a non-equilibrium glass relaxation. The network rapidly descends into a "memorization basin" (low-loss, low-entropy state), from which it slowly relaxes into a high-entropy, generalizing basin. There is no entropy barrier in the transition—the process is barrier-free relaxation rather than a first-order phase transition (Zhang et al., 16 May 2025).
5. Robustness and Regularization: Decay of the $\ell_2$ weight norm increases the "radius of robustness," causing the model's predictions to become stable under perturbation and pulling test points inside the effective decision margin, which triggers sudden generalization. Input/noise augmentation and explicit group invariance regularizers (e.g., commutativity in modular addition) can "de-grok" (accelerate) this transition (Tan et al., 2023).
6. Spectral Bottlenecks and Optimizer Effects: Grokking plateaus arise from extreme spectral imbalance in gradient dynamics—fast modes (principal directions) are learned quickly, but "slow modes" critical for generalization evolve orders of magnitude more slowly. Modifying the optimizer to equalize spectral speeds (e.g., Egalitarian Gradient Descent) or amplify slow gradient components (Grokfast algorithm) can dramatically shorten or eliminate grokking delays (Pasand et al., 6 Oct 2025, Lee et al., 2024).
7. Phase Transition Perspective: Statistical mechanics approaches model grokking as a first-order phase transition, with order parameters describing feature alignment or kernel eigenstructure crossing a critical point, leading to emergent generalization in a mixed-feature phase (Rubin et al., 2023).
8. Statistical and Data Distribution View: Grokking is associated with a distribution shift between training and test datasets, most starkly visible under sub-category imbalance or missing subclasses. Generalization is delayed until regularization (direct or implicit) forces the boundary to align with under-represented or absent test regions (Carvalho et al., 3 Feb 2025).

3. Experimental Paradigms and Diagnostic Metrics

Empirical studies span tasks and architectures:

• Algorithmic: Modular addition/multiplication, parity, group composition, XOR. Highly controlled, favor delayed generalization (Mohamadi et al., 2024, Rubin et al., 2023, Levi et al., 2023).
• Real-World Datasets: MNIST, CIFAR-10/100, Imagenette, QM9, IMDb (Fan et al., 2024, Humayun et al., 2024, Lee et al., 2024).
• Benchmarks: Small subsets of data, biased sampling, or deliberate class imbalance to trigger distribution shifts (Carvalho et al., 3 Feb 2025).

Quantitative and diagnostic tools include:

Metric	Purpose/Insight	Typical Usage
LMN, Rate–Distortion	Model intrinsic complexity/compression	Tracks the compression phase and predicts grokking (Liu et al., 2023, DeMoss et al., 2024)
Local Complexity $C_{\mathrm{loc}}$	Spline-region density / partition geometry	Diagnoses grokking, delayed robustness, and region migration in input space (Humayun et al., 2024)
Robustness metrics (PE/PMI, MID/ED)	Predict grokking, quantify input–output stability	Early indicators, can accelerate transition (Tan et al., 2023)
NTK/Feature Covariance Rotation	Marks kernel–to–feature transitions	Direct visualization of representation learning (Mohamadi et al., 2024, Lyle et al., 26 Jul 2025)
Spectral Oscillation (Fourier loss curve)	Early prediction of grokking	Allows pruning hyperparameter search (Notsawo et al., 2023)
Sharpness/Gap Parametrization	Quantifies sharpness of accuracy transition	Enables comparison of grokking gap vs. transition sharpness (Miller et al., 2024)
Feature Rank Collapse	Detects multistage generalization, tunnel effect	Superior to weight norm as predictor in deep MLPs (Fan et al., 2024)

4. Algorithmic and Optimization Strategies for Accelerating or Eliminating Grokking

The slow-to-emerge generalization of grokking can be mitigated or eliminated via several algorithmic interventions:

• Spectral Equalization/Gradient Manipulation: Egalitarian GD or Grokfast precondition the gradient, amplifying slow directions to equalize learning speeds and collapse plateaus. These approaches excel in parity, modular arithmetic, and other settings sensitive to gradient spectrum (Pasand et al., 6 Oct 2025, Lee et al., 2024).
• High-Entropy–Seeking Optimizers: WanD (Wang–Landau MD in parameter space) samples parameter regions of high entropy at fixed loss, favoring generalizing solutions and bypassing the delayed glassy relaxation phase (Zhang et al., 16 May 2025).
• Norm Regularization and Robustness Augmentation: Weight decay or explicit robustness induction (Gaussian input noise, Jacobian/Lipschitz penalties, group-theory regularizers) can accelerate grokking, often by enforcing the necessary algebraic invariances to support generalization (Tan et al., 2023).
• Numerical Remedies: StableMax activation and perpendicular-gradient optimizers prevent softmax collapse and naive loss minimization, confronting the numerical "edge-of-stability" issues that can prevent generalization without regularization (Prieto et al., 8 Jan 2025).
• Knowledge Distillation: Transferring knowledge from a model that has already grokked on a related (or even different) distribution can induce grokking in data-scarce or distribution-shifted settings, reducing the critical data threshold and mitigating catastrophic forgetting in continual learning (Singh et al., 6 Nov 2025).
• ELR Re-Warming: Periodic effective learning-rate resets (e.g., Normalize-and-Project or cyclical learning rate schedules) re-initiate rich feature-learning, enabling on-demand grokking and overcoming primacy bias in online or nonstationary tasks (Lyle et al., 26 Jul 2025).

5. Connections to Complexity, Compression, and Generalization Theory

Grokking is now firmly situated at the intersection of dynamic model complexity, implicit regularization, and information-theoretic compression. The consensus across approaches is that:

• Early solutions after training set interpolation are high complexity (e.g., large LMN, high intrinsic code length), corresponding to lookup-table like memorization.
• The compression phase, sometimes observed as a double-descent in feature rank or complexity, culminates in the sudden collapse to a low-rank, low-entropy, highly compressible representation—a precondition for robust generalization (Liu et al., 2023, DeMoss et al., 2024, Fan et al., 2024).
• Explicit rate–distortion or MDL principles can yield PAC-Bayes–style generalization guarantees tied closely to empirical complexity estimates (DeMoss et al., 2024).

Notably, in fully linear or kernelized settings, grokking can arise purely as a measurement artifact: smooth loss curves pass a non-linear accuracy threshold later for test than for train data, without any change in the solution's nature (Levi et al., 2023).

6. Practical Guidance, Limitations, and Open Questions

Practical recommendations include:

• Early detection: Monitor information-theoretic, robustness, or complexity metrics (e.g., PE/PMI, LMN, feature rank) to predict and detect grokking transitions earlier than test accuracy would reveal (Tan et al., 2023, Liu et al., 2023, Fan et al., 2024).
• Avoidance vs. Exploitation: To eliminate undesirable grokking plateaus, use footprint-efficient optimizers, high-entropy trajectory sampling, or explicit regularization; to harness grokking for structured algorithmic learning, maintain moderate data sparsity, small weight decay, and allow delayed onset (Pasand et al., 6 Oct 2025, Lee et al., 2024).
• Complexity regularization: Spectral entropy regularizers systematically induce easier compression and more predictable grokking (DeMoss et al., 2024).
• Continual learning and transfer: Knowledge distillation and re-warming protocols overcome critical data thresholds, enable transfer, and prevent forgetting (Singh et al., 6 Nov 2025, Lyle et al., 26 Jul 2025).
• Statistical phenomenon, not just sparsity: Grokking is fundamentally tied to distribution shift, not just data sparsity or high regularization; richness or explicit equivariance in class structure can substitute for sample presence in accelerating grokking (Carvalho et al., 3 Feb 2025).

Open questions encompass:

• The universality and precise criticality of observed phase transitions—barrier-free (glassy) or sharp (first-order);
• The minimal sufficient conditions for grokking in high-data, low-regularization, or practical (non-algorithmic) domains;
• The interactions between numerical phenomena (edge-of-stability, floating-point effects) and representational learning;
• Whether more powerful hybrid regularization, optimizer, or architectural methods can arbitrarily schedule or eliminate grokking;
• The optimality of sharpness/gap metrics for comparing grokking across diverse models and tasks (Miller et al., 2024).

Grokking remains a central diagnostic and theoretical tool for interrogating the transition from memorization to generalization in deep networks, with implications for representation learning, optimization, robustness, and complexity theory across the entire spectrum of modern machine learning.