Symmetry in language statistics shapes the geometry of model representations

Abstract: Although learned representations underlie neural networks' success, their fundamental properties remain poorly understood. A striking example is the emergence of simple geometric structures in LLM representations: for example, calendar months organize into a circle, years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded by a linear probe. We show that the statistics of language exhibit a translation symmetry -- e.g., the co-occurrence probability of two months depends only on the time interval between them -- and we prove that the latter governs the aforementioned geometric structures in high-dimensional word embedding models. Moreover, we find that these structures persist even when the co-occurrence statistics are strongly perturbed (for example, by removing all sentences in which two months appear together) and at moderate embedding dimension. We show that this robustness naturally emerges if the co-occurrence statistics are collectively controlled by an underlying continuous latent variable. We empirically validate this theoretical framework in word embedding models, text embedding models, and LLMs.

• The paper demonstrates that translation symmetry in token co-occurrence statistics induces predictable geometric manifolds, such as circles and curves, in word and sentence embeddings.
• It employs spectral analysis and eigendecomposition to link discrete Fourier modes with embedding geometry, revealing robustness through collective latent variable effects.
• Empirical results across diverse architectures validate the theory, highlighting accurate linear decoding and resilient embedding structures even under significant data perturbations.

Symmetry in Language Statistics as the Origin of Geometric Structures in Model Representations

Overview and Motivation

The paper "Symmetry in language statistics shapes the geometry of model representations" (2602.15029) presents an in-depth theoretical and empirical analysis of how translation symmetries in the co-occurrence statistics of language tokens induce low-dimensional geometric manifolds in word and sentence representations. This work provides a unified organizing principle explaining the emergence of structured embedding geometries—such as circles for cyclic concepts and one-dimensional curves for continuums—in both shallow word embedding models and deep LLMs.

The central claim is that when the co-occurrence probability between pairs of tokens (e.g., months, years, or US states) depends only on their relative “distance” in an underlying semantic continuum, the resulting embedding geometry aligns with Fourier modes in the corresponding space. This geometric structure is mathematically characterized, shown to be robust to significant perturbations in the original co-occurrence statistics, and demonstrated empirically across multiple architectures.

Theoretical Framework

The core assumption is translation symmetry in the token co-occurrence matrix: for a set of tokens parameterized by coordinates (e.g., months indexed by time of year, states by geographic coordinates), the co-occurrence probability between tokens $i$ and $j$ is a function solely of $\mathrm{dist}(\mathbf{x}_i, \mathbf{x}_j)$. The normalized excess co-occurrence matrix, denoted as $\mathbf{M}^*$, captures these statistics.

A key analytical result relates the eigendecomposition of $\mathbf{M}^*$ to the principal component representation of the learned embeddings. For periodic concepts—such as calendar months—$\mathbf{M}^*$ is approximately circulant, and its eigenvectors are discrete Fourier modes. This constrains token embeddings to lie on geometric manifolds like circles or Lissajous curves, with mode amplitudes dictated by the co-occurrence kernel’s spectrum.

Empirical validation of this theory is provided in (Figure 1), which shows principal component projections and Gram matrices of token embeddings for calendar months and historical years, at both the level of static word embeddings and internal LLM residual stream activations.

Figure 1: Cyclic representation manifolds and corresponding Gram matrices for months and years, predicted by analytical theory, observed in word embeddings, and in LLMs.

For open intervals, such as years or spatial sequences with boundaries, the kernel $\mathbf{M}^*$ is Toeplitz, leading to different but analytically predictable embedding geometries—open “rippling” curves whose principal components are also sinusoidal but subject to non-periodic boundary conditions.

Figure 2: Embedding projections for years form Lissajous curves (left) as predicted; right—train/test error scaling for linear decoding of temporal coordinates from embeddings.

The theoretical analysis extends to higher-dimensional semantic spaces, such as geography, where the co-occurrence kernel depends on Euclidean distance on the sphere, inducing slowly-varying top principal components that reflect spatial mapping.

Figure 3: Top PCA modes for embeddings of US states, demonstrating agreement between theoretical kernel-derived modes, word embeddings, text embeddings, and LLM representations.

Robustness and Collective Phenomena

A central empirical finding is that these geometric structures are robust to significant perturbations in the token co-occurrence statistics, even when all direct co-occurrences among a set of tokens (e.g., month names) are ablated. The geometry persists at moderate embedding dimensions, contrary to the naive expectation that structure should break down without direct co-occurrence signals.

This phenomenon is explained via a collective latent variable model, whereby a “seasonality” (for months) or other continuous attribute modulates co-occurrence statistics for a broad set of helper tokens—not just within the focused token subset. Through spectral dominance in the PMI or $\mathbf{M}^*$ matrix, a low-dimensional structure emerges that is insensitive to localized perturbations or ablations.

Figure 4: Circular month embeddings persist after ablating all direct month-month co-occurrences; helper words with high seasonality restore correct ordering.

These results are further validated and contrasted for seasonal vs. non-seasonal helper words, demonstrating that only latent variable-aligned token populations enable the geometric manifold’s reconstruction.

Quantitative and Empirical Results

• Linear Decodability: The top principal components of learned representations for temporally or spatially ordered concepts enable accurate linear decoding of underlying coordinates. The paper predicts and confirms the scaling of decoding error with the number of principal components, finding that the error decays polynomially in the number of components, tightly matching theory (Figure 2, right).
• Manifold Visualization: Embedding projections for cyclic concepts (months) form circular or “Pringle”-shaped manifolds (Figure 1, Figure 5), while open sequences (years) trace explicitly predicted Lissajous curves (Figures 1, 2).
• US Geography: Gram matrices and top PCA directions in embeddings reflect smooth spatial variation, with model-derived kernels capturing the semantic continuum despite real-world non-lattice irregularity (Figure 3, Figure 6).
• Helper Reconstruction: Progressive addition of seasonal helper words monotonically reduces the error in month manifold reconstruction, following predicted scaling (Figure 7, Figure 8).
• Spectral Analysis: The empirical spectra of $\mathbf{M}^*$ and PMI matrices demonstrate the presence of large, isolated eigenvalues associated with latent variables, explaining robustness phenomena (Figure 9).

Implications and Future Directions

Practical implications include principled architectural and data-centric strategies for inducing interpretable and robust geometric structures in embedding spaces, facilitating linear decoding, analogy-making, and compositionality. The collective latent variable mechanism explains why even sparse or indirect statistical signals can suffice for robust feature geometry, reconciling prior findings on analogy persistence and data ablation.

From a theoretical perspective, this work provides a unifying principle underpinning diverse empirical phenomena observed in both word embeddings and LLMs—tying representational geometry back to fundamental distributional statistics and symmetry.

Connections to neuroscience are discussed, as grid cells in the entorhinal cortex similarly encode position via Fourier-like interference patterns, implying that learned spatial representations in animals could arise from analogous symmetry-driven latent variable inference.

Limitations are acknowledged: the primary theory applies most directly in the word embedding matrix factorization regime; contextual adaptation, polysemy disambiguation, and compositional or hierarchical attributes in LLMs remain open for theoretical treatment.

Conclusion

This paper establishes that translation symmetries in pairwise language statistics are the fundamental drivers of geometric structure in learned representation spaces—predicting and empirically validating the form of cyclic, linear, and higher-dimensional manifolds in both shallow and deep networks. The discovered robustness of these manifolds to direct co-occurrence ablation, explained via collective low-rank effects induced by continuous latent variables, has broad consequences for language modeling, interpretability, and cognitive modeling.

Future developments are likely to explore extensions to hierarchical and compositional concept embeddings, nonlinear adaptation in context-aware models, and potential applications in designing more transparent and controllable representation learning architectures.