Manifold Generalization Provably Proceeds Memorization in Diffusion Models

Abstract: Diffusion models often generate novel samples even when the learned score is only \emph{coarse} -- a phenomenon not accounted for by the standard view of diffusion training as density estimation. In this paper, we show that, under the \emph{manifold hypothesis}, this behavior can instead be explained by coarse scores capturing the \emph{geometry} of the data while discarding the fine-scale distributional structure of the population measure~$μ{\scriptscriptstyle\mathrm{data}}$. Concretely, whereas estimating the full data distribution $μ{\scriptscriptstyle\mathrm{data}}$ supported on a $k$-dimensional manifold is known to require the classical minimax rate $\tilde{\mathcal{O}}(N^{-1/k})$, we prove that diffusion models trained with coarse scores can exploit the \emph{regularity of the manifold support} and attain a near-parametric rate toward a \emph{different} target distribution. This target distribution has density uniformly comparable to that of~$μ{\scriptscriptstyle\mathrm{data}}$ throughout any $\tilde{\mathcal{O}}\bigl(N^{-β/(4k)}\bigr)$-neighborhood of the manifold, where $β$ denotes the manifold regularity. Our guarantees therefore depend only on the smoothness of the underlying support, and are especially favorable when the data density itself is irregular, for instance non-differentiable. In particular, when the manifold is sufficiently smooth, we obtain that \emph{generalization} -- formalized as the ability to generate novel, high-fidelity samples -- occurs at a statistical rate strictly faster than that required to estimate the full population distribution~$μ{\scriptscriptstyle\mathrm{data}}$.

• The paper demonstrates that manifold recovery in diffusion models strictly outpaces density estimation by leveraging dominant local geometric structure.
• It introduces a novel δ-coverage criterion with minimax-optimal rates for recovering manifold projections, depending on the manifold's smoothness β.
• The findings reconcile empirical discrepancies and point to new architectures and training criteria that enhance privacy and data augmentation.

Manifold Generalization Provably Proceeds Memorization in Diffusion Models

Motivation and Problem Setting

Diffusion models deliver high-dimensional generative performance but exhibit a puzzling empirical fact: genuinely novel samples emerge predominantly during early training stages or under restricted score network capacity, when the learned score is notably inaccurate. This contradicts classical diffusion theory, which treats score learning as density estimation, predicting monotonic improvement in data fidelity as the score estimation improves. The paper investigates this discrepancy and formalizes the phenomenon under the manifold hypothesis: data is supported on a low-dimensional $k$-dimensional $C^\beta$ submanifold $M \subset \mathbb{R}^D$ ($k \ll D$). The operational target is not minimax density recovery but uniform coverage of the manifold at nontrivial spatial resolution.

Coverage and Statistical Rate Separation

The central technical contribution is a sharp statistical separation between geometry learning (manifold recovery) and density estimation. The authors introduce a coverage criterion: a distribution $\mu$ has $\delta$-coverage of $M$ if it assigns mass comparable to the population measure to every geodesic ball $B_M(y, \delta)$ ($y \in M$). For empirical measures supported on $N$ points, the finest possible resolution is $\tilde O(N^{-1/k})$, the classical minimax rate. However, the main claim is that diffusion models, trained only to coarse score accuracy, achieve coverage at much finer scale $\delta = \tilde O(N^{-\beta/(4k)})$, depending only on manifold smoothness $\beta$.

Geometry Dominates Distribution in Score-Based Models

The core insight is the geometric expansion of the small-noise score function:

$$s^\star(x, t) = -\frac{x - \mathrm{Proj}_M(x)}{t} + \nabla_M \log p(\mathrm{Proj}_M(x)) + \frac{1}{2} H(x) + o(1)$$

where $p$ is the on-manifold density, $H$ mean curvature, and $\mathrm{Proj}_M$ is the nearest-point projection. The leading term $-(x-\mathrm{Proj}_M(x))/t$ dominates for small noise, rendering geometry (the manifold's projection structure) much easier to recover than the full population density. Subsequently, manifold recovery (and coverage) proceeds strictly faster than density estimation, especially in settings of irregular $p$.

Theoretical Guarantees and Algorithmic Realization

The hybrid sampler (reverse-time SDE down to $t_0$, then probability-flow ODE to $\tau$) reflects practical diffusion implementations. For sufficiently coarse scores, the terminal ODE stage essentially learns an approximate projection map. The main theorem asserts minimax-optimal Hausdorff recovery of $M$ and uniform projection accuracy:

$$d_\mathrm{Haus}(M, \hat M) = \tilde{O}(N^{-\beta/k}), \quad \|\mathrm{Proj}_M - \hat{\mathrm{Proj}}\|_\infty = \tilde{O}(N^{-\beta/(2k)})$$

The resultant output distribution achieves $\delta$-coverage at scale $\delta = \tilde{O}(N^{-\beta/(4k)})$. Notably, this rate is strictly superior for large $\beta$ compared to classical density estimation.

Figure 1: Manifold error drops rapidly while memorization rate stays low for coarsely optimized scores, indicating geometry learning precedes memorization; mean alignment demonstrates early recovery of projection geometry.

Function Classes and Local Representation

A rigorous characterization is provided for the function class underlying score recovery. Via the eikonal equation and local graph representation, member functions are shown to be distance-like, aligning the score network's implicit bias with projection geometry. The manifold's local chart admits a representation as a graph over the tangent space:

Figure 2: A local representation of a submanifold $M \in C^\beta$ by graphing a function over its tangent space neighborhood.

Both theoretical and analytic results ensure that, under mild regularity (reach and bounded derivatives), the learned score class induces smooth projection maps. The change of basis between hypothesis and ground-truth manifold is formalized with explicit bounds on derivatives:

Figure 3: Visualization of the hypothesis score function class $\{s_\eta\}$ in local coordinates, illustrating unique projection and tangent representations.

Figure 4: Change of basis for any $\hat x$ in $M$; coordinates in the hypothesis and ground truth basis are linked via invertible transformation.

Connection to Existing Literature and Contradictory Claims

The paper's statistical results contradict the standard paradigm where density recovery is necessary for generalization. Existing minimax manifold estimation frameworks (e.g., [AamariLevrard2019]) achieve similar rates for geometric recovery, but are fully nonparametric and do not provide coverage guarantees for diffusion models. Moreover, empirical literature on memorization mitigation and geometric diagnostics ([SSG23], [GDP23], [achilli2025memorization], [kadkhodaie2023generalization]) is reconciled via the geometric separation principle elucidated herein.

Practical and Theoretical Implications

Practically, the findings suggest that diffusion models can be reliably used to produce novel samples with strong manifold fidelity well before memorization or density fitting occurs, bolstering their utility in privacy and data augmentation scenarios. Theoretically, the separation between generalization and memorization calls for new architectural and training-criterion designs aligned with geometric inductive bias, potentially realized via physics-informed neural networks or modified score-matching objectives.

Future Directions

Several research avenues are laid out:

• Transition from nonparametric to parametric realization of projection-like score functions, possibly via PINNs.
• Extending guarantees to more general noise schedules and discrete-time samplers.
• Linking coverage metrics to perceptual quality and task-level novelty.
• Precise optimization of constants and dependencies in the main rates.

  Figure 5: Compact local representation of a submanifold, elucidating intrinsic distance and geometric structure critical for manifold estimation.

Conclusion

The analysis establishes that, under the manifold hypothesis, generalization in diffusion models is both provable and precedes memorization, mediated via fast geometric recovery of support rather than rigorous density estimation. This statistical and algorithmic separation motivates further developments in generative modeling that exploit data geometry, offering improved sample diversity and robustness in practice.

---

Reference: "Manifold Generalization Provably Proceeds Memorization in Diffusion Models" (2603.23792)