SeeSE3: Emergence of 3D Space in Vision Features

Abstract: In this paper, we ask whether vision foundation models construct representations that reflect the intrinsic properties of 3D Euclidean space. Unlike previous works that probe 3D awareness of vision features by regressing image-centric quantities such as depth or normals, we investigate the relation between the structure of the space of visual features and the group of Euclidean transformations $SE(3)$. We propose a set of probes to evaluate this relation from both topological and geometric perspectives: a mutual neighborhood metric that measures the alignment between feature neighborhoods and spatial topology, and a Poincaré Adapter to test the linear accessibility of the geometry of camera motion from latent displacements in static scenes. We show that self-supervised vision models, which, in principle, have not been trained with direct 3D supervision or active agency, possess latent subspaces that are remarkably strongly correlated with three-dimensional Euclidean space, when probed correctly. Building on this insight we propose a new class of "Latent-Space Navigation" techniques that perform visual odometry and localization purely in the latent space, bypassing the need for explicit 3D reconstruction.

• The paper demonstrates that self-supervised vision encoders naturally organize latent features to reflect the 3D SE(3) structure.
• The paper introduces a lightweight Poincaré Adapter that linearizes feature trajectories, facilitating accurate pose displacement estimation.
• The paper shows that increased data scale and dense scene structure boost topological alignment, supporting robust latent space navigation for robotics.

Emergence of 3D Euclidean Structure in Self-Supervised Vision Models

Introduction and Problem Formulation

"SeeSE3: Emergence of 3D Space in Vision Features" (2607.14228) addresses the question of whether representations learned by vision foundation models encode the geometry of 3D Euclidean space—specifically, whether latent feature spaces structured by large vision encoders reflect the algebraic structure of $SE(3)$ (the group of rigid-body motions). Distinct from prior works that probe for monocular depth or surface-normal awareness, this paper evaluates alignment between feature space topology/geometry and the underlying 6D Euclidean spatial manifold, independent of explicit 3D supervision or agency.

The motivating hypothesis, inspired by Poincaré, posits that even "motionless observers" (vision encoders trained passively) might discover the structure of space via high-dimensional feature organization, challenging the classical assertion that agency is a prerequisite for spatial reasoning.

Figure 1: Vision features are analyzed for alignment with Euclidean transformations; a simple lightweight "Poincaré adapter" can linearize tangled feature trajectories to closely track the camera’s $SE(3)$ path.

Methodology: Metrics and Probing Geometry

The central methodological innovation is the introduction of hierarchical probes:

• M1: Topological Alignment: The mutual $k$-nearest-neighbor (kNN) alignment metric compares local neighborhood structures in the learned feature space and the $SE(3)$ pose manifold. High alignment indicates feature adjacency reflects spatial adjacency.

  Figure 2: Mutual kNN alignment between feature and pose spaces at short and long strides discriminates pixel-level versus spatial-level awareness.

• M2: Intrinsic Dimensionality (ID): Using estimators (e.g., Two-NN, MLE), the local dimensionality of the latent feature trajectory in static scenes is measured. ID close to 6 suggests encoding of all $SE(3)$ degrees of freedom.
• M3: Linear Equivariance: Direct linear regression testing whether differences in feature space $(z_{t+s} - z_t)$ can be mapped to Lie algebra pose displacements ($\Delta P$), i.e., $\Delta P \approx W(z_{t+s} - z_t)$.
• M4: Poincaré Adapter: A lightweight Siamese MLP nonlinearly transforms features to maximize local linearity between pose displacement and latent difference, isolating the latent subspace most aligned to spatial motion.

Empirical Results

1. Topological and Geometric Alignment

Explicit geometric models (DUSt3R, MoGe) achieve the best global alignment scores on the mutual kNN metric, confirming the hypothesis that geometric pretext tasks structure feature spaces to reflect spatial adjacency. Surprisingly, purely self-supervised models (notably the DINO family) exhibit strong emergent topological alignment purely from passive observation, with DINOv2 achieving kNN scores comparable to geometric models.

Video models and raw pixels perform dramatically worse, implicating the importance of static dense scene structure rather than generic temporal motion cues.

2. Intrinsic Dimensionality and Linear Decodability

Although many feature spaces exhibit an intrinsic dimensionality close to 6, all models (except for explicitly geometric ones) yield negative or near-zero $R^2$ for direct linear mapping from feature differences to $SE(3)$ displacements. This demonstrates that the raw latent manifold is inherently non-Euclidean and curved, not natively supporting vector arithmetic for pose displacement estimation.

3. Emergence of Linear Structure via Adapter

With the introduction of the Poincaré Adapter, DINOv2 and other strong models achieve test-set $SE(3)$0 for linear decoding of $SE(3)$1 displacements, indicating the presence of a robust locally Euclidean submanifold. This result is robust across datasets, with generalization performance scaling—larger and more diverse training environments yield higher decodability and broader transferability (Figure 3).

Figure 3: Zero-shot pose recovery accuracy grows monotonically with the number of training environments, with no observed saturation at $SE(3)$2.

4. Latent Space Navigation

Using only the vector arithmetic defined by the adapted latent space, the system enables closed-form "latent space navigation"—generating representations of future frames corresponding to arbitrary pose displacements without explicit 3D reconstruction:

Figure 4: Multi-step open-loop latent navigation over four steps ($SE(3)$3 m) remains visually and spatially coherent, demonstrating applicability for latent planning.

Empirically, a simple pseudo-inverse-based navigator outperforms learned deep correctors on topological retrieval, confirming the sufficiency of linear dynamics in the adapted latent space for accurate spatial navigation.

Figure 5: Multi-step navigation trajectories of various lengths—perfect recovery for small displacements, small drift for moderate, and precise rotation even with large translation.

Theoretical Analysis and Ablations

A formal analysis establishes that any smooth encoder whose Jacobian with respect to $SE(3)$4 has full rank admits a local linear readout for $SE(3)$5 displacement. However, "curvature"—the variability of the Jacobian across scenes—prevents a global, single linear map from universally decoding pose. The nonlinear Poincaré Adapter effectively "untangles" this curvature, yielding a flat coordinate patch up to a group-theoretic residual inherent to $SE(3)$6 composition, which vanishes for pure translation.

The difficulty in decodability of component motions is analytically justified: rotation recovery is almost always reliable (depth-independent), whereas translation, dependent on scene depth, is unreliable for magnitude absent explicit scale information.

Ablation studies confirm:

• Stable mutual $SE(3)$7-nn alignment (topology) arises easily across training recipes.
• Geometric decodability is fragile, highly sensitive to data scale, hyperparameter stochasticity, and especially to self-distillation and batch size (with topology-geometry decoupling across seeds).
• Absolute pose regression from features is entirely ineffective; only relative, locally linearized metrics expose the spatial structure.

Implications for AI, Robotics, and Model Design

The findings imply that large-scale, self-supervised vision transformers inherently encode not only high-level semantics but also a robust, decodable representation of 3D Euclidean space without explicit geometric supervision or interaction—a significant structural prior for downstream robotics, SLAM, and embodied AI systems.

Practically, this enables latent-space navigation and localization modules that bypass explicit 3D geometric modules, potentially simplifying architectural pipelines for visual navigation in robotics and AR/VR, and provides a novel route for world model design where navigation and spatial reasoning occur in high-level feature space.

Theoretically, the work supports the existence of abstract "visual grid codes" in the absence of physical agency, linking vision model geometry to neural representations studied in biological systems, and prompts a reevaluation of agency-centric views of spatial cognition in machine learning.

Conclusion

This paper demonstrates that self-supervised vision encoders—without access to depth, pose, or interactive signals—organize their latent features to reflect the geometry and topology of $SE(3)$8. While the raw representation is highly nonlinear, a simple MLP adapter is sufficient to reveal a locally linear, homogeneous submanifold supporting vector arithmetic for 6DoF motion. Topological alignment emerges robustly, while geometric (metric) equivalence requires data scale and is stochastic. The latent space, when probed and adapted correctly, supports closed-form navigation and spatial reasoning applicable to vision-based control and robotics.

Future work should investigate extension to dynamic scenes, where "changes of position" and "changes of state" are entangled by underlying physical processes, and explore integration into end-to-end differentiable world model architectures.

(2607.14228)