Geometry Foundation Model

• Geometry Foundation Models are universal pretrained architectures that use non-Euclidean and topological structures to represent complex data such as graphs, manifolds, and 3D scenes.
• They employ tailored geometric operations like exponential/log maps and parallel transport to align structural patterns across diverse data domains for effective transfer learning.
• By utilizing a shared vocabulary of geometric tokens, these models achieve superior expressivity and parameter efficiency compared to traditional Euclidean approaches.

A Geometry Foundation Model (GFM) refers to a universal, pretrained model whose architecture, training strategy, and inductive biases are tailored to the underlying geometric and topological properties of data. Unlike standard deep neural networks operating purely in Euclidean space, GFMs leverage Riemannian geometry, non-Euclidean manifolds, or algebraic geometric structures to induce generalization across disparate domains—spanning graphs, point clouds, 3D vision, and relational knowledge bases. The foundation model approach organizes learning around a shared vocabulary of geometric objects (subgraphs, submanifolds, spatial relations), with the aim of cross-domain and zero-shot transfer.

1. Motivation: Why Geometry in Foundation Models

The shift from Euclidean to geometric foundation models is motivated by both theoretical and empirical limitations of standard approaches. Real-world data—such as graphs, manifolds, and 3D scenes—exhibit non-flat, hierarchical, and cyclic structures that are poorly modeled in Euclidean latent spaces. Classical theorems (e.g., Matoušek’s embedding lower bounds, Markov p-convexity) demonstrate high distortion and exponential dimension requirements when embedding trees or cycles into flat spaces, while empirical evidence shows that hyperbolic or spherical coordinates capture hierarchical and cyclical structures with much lower distortion (He et al., 11 Apr 2025). Distinct tasks—hierarchical representations, cyclic/rotational invariance, hierarchy in graphs—naturally select different geometries:

Data/Task Type	Best-Fit Geometry	Euclidean Distortion
Hierarchy, trees	Hyperbolic (K<0)	High
Cyclic structures	Spherical (K>0)	High
Mixed motifs, graphs	Product bundles	Intermediate

Empirical studies further confirm that token/graph embeddings in LLMs or GNNs are often more hyperbolic than Euclidean (He et al., 11 Apr 2025).

2. Geometric Architecture: Riemannian and Algebraic Designs

A geometry foundation model is defined by its architectural adaptation to non-Euclidean geometric domains:

• Product bundle manifolds: Foundation models such as RiemannGFM construct their latent space as a product of constant-curvature manifolds (e.g., hyperbolic H^d_κ, spherical S^d_κ) and their tangent bundles (Sun et al., 5 Feb 2025). Each factor targets a motif: tree fragments embed in hyperbolic, cycles in spherical geometry.
• Coordinate and tangent-space representations: Each node or token is associated both with a manifold coordinate (p ∈ M) and a tangent space encoding (z ∈ T_pM), with updates that are geometry-aware (using the appropriate exponential/log maps, parallel transport, and Riemannian metrics).
• Algebraic/geometric message passing: For knowledge graphs, gamma-type models introduce multi-head geometric attention, leveraging real, complex, split-complex, and dual transformations to model symmetric, cyclic, hierarchical, and translational patterns, respectively (Xin et al., 28 Dec 2025).

Non-Euclidean attention, product-manifold feature fusion, and contrastive learning between geometric views (e.g., hyperbolic vs. spherical embeddings) are core mechanisms (Sun et al., 5 Feb 2025, Xin et al., 28 Dec 2025).

3. Structural Vocabulary and Universal Building Blocks

Structural vocabulary is central to geometric foundation modeling. RiemannGFM, for example, defines its universal building blocks as a small, shared set of tree fragments and cycles. Any connected graph can, in principle, be assembled from overlapping instances of these substructures (Sun et al., 5 Feb 2025). These are extracted as follows:

• Tree tokens: Rooted trees of bounded depth (e.g., breadth-first subtrees).
• Cycle tokens: Simple cycles up to a designated length (triangle, quadrangle, ...).

This vocabulary functions analogously to word tokens in LLMs, supporting transfer learning across previously unseen graphs or domains. The geometry of each token type determines its embedding manifold (tree-like → hyperbolic; cycle-like → spherical).

4. Pretraining, Objectives, and Cross-Domain Transfer

Geometric foundation models are typically pretrained in a task-agnostic, self-supervised regime to maximize cross-domain generalization:

• Self-supervised contrastive objectives: Models such as RiemannGFM use the dual geometric views of each node (e.g., hyperbolic and spherical tangent-space encodings) as positive pairs in a contrastive loss, aligning representations that capture complementary aspects of structure (Sun et al., 5 Feb 2025).
• Vocabulary-guided updates: Learning alternates between structural coordinate updates (e.g., cross-geometry attention to align substructure nodes) and global encoding updates (e.g., bundle convolution with parallel transport).
• No reliance on textual/attribute labels: Structural patterns are exploited rather than semantic labels or text attributes.
• Transfer mechanisms: The shared backbone and vocabulary generalize across a range of downstream tasks (node classification, link prediction, zero-shot transfer) and disparate domains (academic, code, infrastructure, etc.).

Empirical evaluations demonstrate that such pretraining achieves stable, often superior, performance compared to Euclidean baselines, language-aligned graph models, and self-supervised alternatives over both homophilic and heterophilic datasets (Sun et al., 5 Feb 2025).

5. Geometric Model Expressivity and Adaptivity

Adopting geometric spaces yields expressivity not attainable in purely Euclidean models:

• Expressive function classes: Multi-algebra architectures (e.g., Gamma) have strictly greater capacity than any constituent space alone, as no single algebra supports all observed relation types (symmetric, anti-symmetric, hierarchical, one-to-many) (Xin et al., 28 Dec 2025).
• Mixed-geometry modules: Product-manifold and mixture-of-experts designs dynamically route data or sub-structures through the geometry best suited to their local topology (e.g., Gromov δ-hyperbolicity prompts, product bundles; (Sun et al., 5 Feb 2025, He et al., 11 Apr 2025)).
• Learnable curvature and adaptivity: Models can make curvature a learnable parameter, adapting the geometry across tasks or data regions (He et al., 11 Apr 2025).

The ability to dynamically configure embedding geometry or to combine manifold-valued features is highlighted as a necessary ingredient for next-generation scalability and robustness.

6. Mathematical Operations and Implementation Foundations

Geometry foundation models replace standard neural operations with their Riemannian or algebraic analogs:

• Exponential and logarithmic maps: Embeddings are updated via exp/log maps (from/to tangent spaces), respecting the manifold’s curvature (Sun et al., 5 Feb 2025).
• Parallel transport: Gradients, feature vectors, or skip connections require parallel transport along geodesics to ensure geometric consistency when combining features at different loci.
• Geometric attention and aggregation: Attention weights are computed via geodesic distances rather than Euclidean dot products; aggregation applies Fréchet means or weighted exponential sums on the manifold (Sun et al., 5 Feb 2025, He et al., 11 Apr 2025).
• Optimization: Training employs Riemannian optimizers (SGD, Adam), with retraction/projection as needed.

These modifications are mandatory to preserve the geometric structure during forward and backward passes.

7. Outlook: Challenges and Future Directions

Current geometry foundation models demonstrate marked improvements in transferability, parameter efficiency, and expressivity over Euclidean models, but open challenges remain:

• Computational efficiency: Manifold operations incur higher cost. Direct-model methods (e.g., Lorentz-model MLPs) or geometric tensor libraries are under active development (He et al., 11 Apr 2025).
• Dynamic and modular geometry: Adaptive mixing of geometric modules per input or layer is a promising direction.
• Curvature learning and meta-geometry: Discovering optimal curvatures in a data-driven fashion or by meta-learning remains unsolved.
• Benchmarks and metrics: Standardized evaluation of “geometric fidelity” and distortion–performance tradeoff is lacking.
• Extending beyond graphs: Recent results indicate effective geometric transfer to point clouds, 3D vision, knowledge graphs, panoramic and extreme-view 3D perception, and more (Quackenbush et al., 6 Mar 2025, Fang et al., 22 Jul 2025, Zhang et al., 27 Nov 2025, Lin et al., 18 Dec 2025, Xin et al., 28 Dec 2025).

A plausible implication is that widespread adoption of non-Euclidean architectures across all foundation model domains will become standard to overcome the intrinsic limitations of flat geometry (He et al., 11 Apr 2025). The field’s trajectory is toward unified, geometry-consistent models that natively encode the symmetries, hierarchies, and structures observed across complex scientific, relational, and spatial datasets.