Why Self-Supervised Encoders Want to Be Normal

Abstract: Self-supervised learning has achieved remarkable empirical success in learning robust representations without explicit labels, most recently demonstrated within the framework of Joint-Embedding Predictive Architectures (JEPA). However, a fundamental question remains: what analytical principles drive these encoders toward specific distributional states? In this paper, we demonstrate that the preference for normal distributions in self-supervised encoders is a direct consequence of the Information Bottleneck (IB) principle. By recasting the IB objective as a rate-distortion problem over the predictive manifold, we provide a theoretical basis for why optimal, target-neutral, latent representations should tend towards isotropic Gaussian states. Under this framework, we show that latent representations correspond to soft clustering of inputs sharing similar predictive distributions, organized within a natural simplex structure. This perspective unifies a wide range of existing supervised and less-supervised objectives and provides a principled explanation for commonly used regularization schemes. Furthermore, we derive practical loss objectives that approximate this structure and demonstrate their effectiveness on standard benchmarks. Ultimately, our framework offers a geometric lens to understanding representation collapse and it establishes a mathematical system for regularization strategies to be used to ensure high-entropy, informative embeddings in modern self-supervised models.

• The paper shows that self-supervised encoders naturally converge to isotropic Gaussian representations by maximizing entropy on a predictive manifold.
• It reframes the Information Bottleneck objective as a rate-distortion problem, enabling soft clustering of predictive distributions via the CEB framework.
• Empirical results on synthetic and real-world tasks validate that SIGReg effectively enforces high representation entropy and prevents collapse.

Information Bottleneck, Predictive Manifold, and Normality in Self-Supervised Encoders

Overview of the Framework

The paper "Why Self-Supervised Encoders Want to Be Normal" (2604.27743) offers a rigorous information-theoretic and geometric characterization of representation learning via self-supervised encoders, situated within the Information Bottleneck (IB) paradigm. The central thesis is that the preference for isotropic Gaussian latent representations arises naturally from maximum entropy principles applied to the geometry of the predictive manifold. By reframing the IB objective as a rate-distortion problem—with KL divergence quantifying predictive mismatch—the optimal latent variable $W$ must not simply compress $X$ but perform a soft clustering among the predictive distributions $p(Y|x)$, which reside within the canonical probability simplex $\mathcal{P}_K$.

This geometry directly unifies supervised, semi-supervised, and self-supervised objectives and rationalizes recent empirically successful distributional regularization strategies such as SIGReg. The theoretical linkage between the flat Dirichlet prior (maximal entropy on $\mathcal{P}_K$) and the isotropic Gaussian embedding, via a chain of exact transformations, is precise and quantifies the entropy overhead incurred at every stage.

Predictive Manifold and Soft Clustering

The IB methodology formalizes the trade-off between retention of predictive information ($I(W;Y)$) and compression ($I(X;W)$), where a latent variable $W$ is constructed to minimize redundancy and maintain predictive power regarding $Y$. The key insight of the paper is the geometric interpretation: the optimal representation $W$ is always a function of the predictive distribution $X$0, mapping inputs onto the predictive manifold $X$1.

At zero distortion ($X$2), the encoder computes the minimal sufficient statistic, collapsing nuisance variability and achieving the endpoint of the IB curve. For $X$3, the encoder implements soft clustering in $X$4, with each latent $X$5 representing a cluster center in $X$6. This reflects a stochastic assignment where the decoder is always the conditional mean of $X$7 over the preimage of $X$8. The simplex structure ensures invariance to prediction-preserving transformations and bounds the latent dimensionality by $X$9.

Figure 1: Continuous predictive manifold—$p(Y|x)$0 traces a loop through $p(Y|x)$1.

Dirichlet-to-Gaussian Transformation and SIGReg

A critical contribution is the chain of exact transformations connecting the flat Dirichlet prior ($p(Y|x)$2) to an isotropic Gaussian in Euclidean space, thereby explaining the rationale for SIGReg as a Gaussian relaxation of maximum entropy on the simplex. The transformation follows:

1. Flat Dirichlet Prior: Maximum entropy distribution on $p(Y|x)$3.
2. Exponential Parameterization: $p(Y|x)$4 i.i.d. $p(Y|x)$5 random variables, normalized to yield Dirichlet samples.
3. Gaussian Embedding: Each exponential is constructed from the sum of squares of two independent standard normals, giving $p(Y|x)$6 i.i.d. $p(Y|x)$7.

The phase entropy overhead $p(Y|x)$8 nats is strictly non-diminishing and affects rate accounting but not achievable prediction. SIGReg, fitted via the Cramér–Wold theorem and the Epps–Pulley test, matches the encoder’s marginal to $p(Y|x)$9 robustly, enforcing high entropy and preventing collapse.

Practical Losses: CEB and Distributional Regularization

The paper employs the Conditional Entropy Bottleneck (CEB) decomposition for practical estimation, rewriting the IB Lagrangian as conditional rate $\mathcal{P}_K$0 plus total rate $\mathcal{P}_K$1. This allows analytical estimation in supervised settings via minibatch marginals, while in semi-supervised or self-supervised settings, SIGReg serves as the regularizer on unlabeled data.

• Supervised Case: Encoder trained for sufficiency using class-conditional marginals.
• Semi-Supervised Case: CEB conditional rate applied to labeled data, SIGReg penalizes collapse for all data.
• Self-Supervised Case: View-prediction proxy replaces conditional rate, SIGReg regularizes all data.

Empirical Demonstration

The paper empirically validates the framework through synthetic and real-world experiments:

• Continuous Toy Problem: For $\mathcal{P}_K$2 a continuous angle, $\mathcal{P}_K$3 traces a loop in $\mathcal{P}_K$4. Sweeping $\mathcal{P}_K$5 demonstrates the progressive recovery of predictive structure, with rate-distortion curves quantifying trade-offs.

  Figure 2: Dirichlet means vs. ground-truth predictive distributions for increasing $\mathcal{P}_K$6.

  

  Figure 3: Information plane, rate-distortion curve, and $\mathcal{P}_K$7 for the continuous experiment.

• Discrete Toy Problem: For $\mathcal{P}_K$8, 10 predictive clusters are embedded. CEB estimation matches ground-truth clusters, with accuracy saturating as $\mathcal{P}_K$9 reaches the number of classes and rate converging to $\mathcal{P}_K$0.

  Figure 4: Ten distinct $\mathcal{P}_K$1 points on the simplex loop, composing the discrete predictive manifold.

  

  Figure 5: Dirichlet means versus ground-truth for increasing $\mathcal{P}_K$2 in the discrete CEB experiment.

  

  Figure 6: Information plane, rate-distortion curve, and distortion for discrete CEB with dashed lines for $\mathcal{P}_K$3 and $\mathcal{P}_K$4.

• FashionMNIST: Comparison between VIB and non-parametric CEB estimation on the simplex. CEB achieves higher accuracy at lower rate; empirical sweep over $\mathcal{P}_K$5 verifies phase transition and validates theoretical predictions.

  Figure 7: VIB (Gaussian, $\mathcal{P}_K$6) vs. CEB (Dirichlet, $\mathcal{P}_K$7) on FashionMNIST—error rate, information plane, and rate-distortion curves.

• Ablation on Simplex Dimension $\mathcal{P}_K$8: Demonstrates that accuracy saturates at $\mathcal{P}_K$9, with predictive benefit plateauing while distortion continues to decrease as $I(W;Y)$0 increases.

  Figure 8: CEB ablation over simplex dimension $I(W;Y)$1 at $I(W;Y)$2; accuracy saturates at $I(W;Y)$3.

Implications and Future Directions

The framework establishes a geometric and information-theoretic foundation for distributional regularization in less-supervised learning, providing clarity on the circumstances under which latent normality is preferable and the precise entropy overheads incurred. Practically, it supports robust training with high-entropy embeddings, prevents collapse, and enables task-agnostic encoder regularization. The analysis also delineates the structural role of the simplex dimension $I(W;Y)$4 as the sole parameter governing predictive capacity.

Future research directions include:

• Operational strengthening of continuous-$I(W;Y)$5 covering number bounds.
• Empirical evaluation of the exponential parameterization as a near-exact alternative to the Gaussian with diminishing overhead.
• Scaling minibatch-marginal estimators to higher dimensions and larger datasets for non-parametric mutual information estimation.

Conclusion

This paper develops a rigorous geometric perspective on the Information Bottleneck for encoder-decoder networks, identifying why self-supervised encoders gravitate toward isotropic Gaussian representations as a relaxation of maximum entropy on the predictive manifold. The Dirichlet-to-Gaussian chain provides a theoretical footing for SIGReg, while the CEB decomposition enables principled non-parametric encoder losses across varying supervision regimes. The results unify supervised and less-supervised approaches, explain regularization strategies, and empirically validate the geometric theory behind representation collapse and entropy-regularized learning.