Emergence of a Shared Canonical Object Frame from In-the-Wild Videos

Abstract: Comparing object orientations and positions across different instances requires their poses to be expressed in a shared canonical frame. Establishing such frames has traditionally required manual annotation, creating a scaling bottleneck that limits category and instance diversity. We show that a shared canonical frame can instead emerge from self-supervised training on object-centric videos captured in the wild, using only noisy camera poses from Structure-from-Motion. Our key idea is to route all training sequences through a shared geometric bottleneck: a coarse canonical mesh that carries no category-specific detail. By learning dense correspondences from image pixels to this mesh, and estimating per-sequence alignments from noisy SfM geometry, a common canonical frame emerges from multi-view consistency and the semantic priors of the feature extractor, without any canonical pose labels or category conditioning. Trained in a self-supervised manner on 160,000 in-the-wild object videos, our method achieves competitive accuracy on category-level pose estimation benchmarks compared to methods that rely on canonical pose supervision. The code and checkpoint is available on https://github.com/Fischer-Tom/Emergent-Canonical-Frame/.

• The paper demonstrates that a shared canonical frame can emerge from in-the-wild videos using self-supervised learning without explicit pose annotations.
• It introduces a geometric bottleneck with a fixed 3D mesh and a dense correspondence network to enforce multi-view consistency across object categories.
• Experimental results reveal competitive or superior performance to supervised methods on major benchmarks, highlighting scalability and generalized pose estimation.

Emergence of a Shared Canonical Object Frame from In-the-Wild Videos

Motivation and Background

Establishing a shared canonical frame for object pose—the consistent coordinate system across instances and categories—is foundational for applications such as category-level 6D pose estimation, robotics, shape retrieval, and 3D generative modeling. Traditional supervised approaches depend on labor-intensive collection and curation of object pose annotations or aligned synthetic assets, introducing scaling bottlenecks and limiting real-world generalization. Some recent efforts sidestep full supervision via self-supervised pipelines, but these are typically constrained to individual object categories, fail to induce a unified canonical frame, and require explicit category labels or per-category models.

The work "Emergence of a Shared Canonical Object Frame from In-the-Wild Videos" (2606.30058) challenges the prevailing notion that canonical object frames are unattainable without direct supervision or category priors. Instead, it demonstrates that a shared frame can emerge via self-supervised learning from large-scale, object-centric, in-the-wild video data by leveraging only noisy Structure-from-Motion (SfM) geometry and multi-view consistency, dispensing entirely with canonical pose labels and category conditioning.

Figure 1: Comparison of three approaches—supervised methods using pose annotations, self-supervised per-category methods, and the proposed cross-category, canonical-frame learning using a geometric bottleneck.

Methodology

Geometric Bottleneck with a Shared Canonical Mesh

The methodological core is a geometric bottleneck instantiated as a coarse, fixed, category-agnostic 3D mesh (default: unit cube). This primitive defines a coordinate frame shared by all object instances and categories, compelling the learning process to route all pose and appearance information through the same structure, independent of categorical priors.

Dense Correspondence Network

A dense, self-supervised correspondence network predicts distributions over mesh vertices for image pixels. The network receives features from a strong frozen Vision Transformer backbone (DINOv3 ViT-L/16), augmented with LoRA adapters for computational efficiency, and decodes mesh-vertex queries with a transformer decoder. The output is a probabilistic map from pixels to mesh surface locations, from which 2D–3D correspondences in the canonical frame are extracted.

Per-Sequence Alignment and Pseudo-Labeling

Each object video sequence provides an SfM-defined coordinate system, arbitrary up to a pose transformation. The per-sequence alignment module estimates a rigid rotation from the SfM frame to the canonical mesh frame using multi-view, entropy-weighted correspondences. This is initialized by a PCA-based geometric prior (aligned to object elongation) and iteratively refined through co-evolving predictions and pseudo-labels, formulated as a robust Wahba problem solved via SVD. The aligned mesh projections yield dense, per-pixel supervision (pseudo-labels), enforcing consistent correspondences through multi-view geometry.

Figure 3: Method overview—(a) shared canonical mesh, (b) dense correspondence prediction, (c) per-sequence alignment leveraging multi-view information and providing pseudo-label supervision.

Training Objective

The loss combines cross-entropy over vertex correspondences and a mask prediction loss (binary cross-entropy plus Dice), with heightened focus on object boundaries. No category label, canonical pose annotation, or synthetic rendering is used at any stage—the model operates purely on real, unannotated videos.

Inference

At inference, a single RGB image is processed; dense 2D–3D correspondences are produced, and the 6D pose in the canonical frame is recovered via EPnP with RANSAC. The predicted frame is uniquely shared across all instances and categories.

Experimental Evaluation

Cross-Benchmark Evaluation

The model is trained on the UCO3D dataset ($\sim$160k object-centric videos) and evaluated on major category-level pose benchmarks: REAL275, Omni6DPose, Objectron, Pascal3D+, and ImageNet3D. Despite never seeing canonical pose annotations or synthetic data, the method achieves the highest average accuracy across all datasets, outperforming canonically supervised category-agnostic models such as OrientAnything V1/V2 and QWEN3-VL.

Key results: On benchmarks where semantic axes are discriminative (e.g., vehicles, chairs), the method either matches or surpasses supervised competitors. Only in domains with prevalent symmetry (e.g., Omni6DPose) does performance lag, reflecting an inherent limitation of pose ambiguity in symmetric objects without external orientation cues.

Figure 2: Qualitative pose predictions—predicted canonical axes visualized on diverse benchmarks. Semantic “front” and “up” consistently align without explicit supervision.

Category-Agnostic vs. Category-Specific Self-Supervised Baselines

The method is compared to per-category self-supervised baselines (e.g., UOP3D, Common3D), controlling for data scale and distribution. When matched in training data (50 sequences per category), it remains competitive; when scaling up data volume and removing category separation, it surpasses all baselines, demonstrating that data scale and cross-category training are critical for generalizable canonical frame emergence.

Ablations and Analysis

• Per-Sequence Alignment: Ablation demonstrates that PCA initialization alone is insufficient for disambiguating semantic orientation except for highly symmetric objects. Multi-view learned alignment is essential for asymmetric categories.
• Geometric Bottleneck Shape: Cubic mesh induces better semantic region separation than a spherical mesh, but both bottleneck forms enable emergent canonical frames.
• Data Scale and Multi-View Diversity: Increasing sequence and view diversity yields consistent accuracy improvements, confirming scaling as a major driver.
• Symmetry Handling: Main accuracy drop occurs on symmetry-heavy objects (continuous or $n$-fold symmetrical), where orientation ambiguity is inherently ill-posed without explicit symmetry modeling or further sensory cues.

Figure 4: Symmetry-aware accuracy breakdown. Performance gap arises primarily from symmetry-heavy objects, highlighting the challenge of orientation ambiguity absent canonical supervision.

Figure 5: Visualization of per-category convention mappings—most categories cluster into a few dominant modes, confirming the frame’s cross-category consistency.

Figure 7: Accuracy–runtime trade-off. Accuracy saturates at moderate mesh resolutions, enabling practical deployment without substantial efficiency loss.

Implications and Theoretical Insights

The study conclusively demonstrates that emergent canonical frames, useful for pose reasoning and cross-instance comparison, can arise from unconstrained, in-the-wild video and multi-view geometry, provided sufficiently strong feature backbones and geometric bottlenecks. This result has two notable implications:

• Practical Generalization: Eliminating the need for category labels and canonical pose annotations dramatically improves scalability and domain generality, making category-level pose estimation feasible for arbitrary objects and emergent categories.
• Architectural Paradigm: The geometric bottleneck principle should extend to other vision tasks that demand canonicalization or cross-instance correspondence; future work may explore alternative primitives, explicit symmetry modeling, or joint geometry–appearance factorization.

From a theoretical standpoint, the work underscores the potency of multi-view consistency and feature space priors, especially in the absence of direct supervision. The emergence of shared semantic axes without explicit cueing suggests implicit bias in high-capacity vision transformers can be sufficient for structural grounding, provided bottlenecks enforce cross-sequence coupling.

Future Directions

Key future directions include explicit symmetry modeling in the self-supervised framework to represent orientation ambiguity, integrating metric scale prediction (e.g., via monocular depth or learned priors), and extending geometric bottlenecks beyond simple primitives for richer scene-level canonicalization. There are also clear opportunities to apply this paradigm to multi-object or scene-wide canonical frame induction and to jointly optimize detection and pose under the unified frame.

Conclusion

This work presents a principled solution to canonical frame emergence by leveraging object-centric in-the-wild video and geometric bottlenecking rather than annotation or category supervision. Experiments validate that dense, transferable, and consistent canonical frames can be recovered at scale, with competitive or superior accuracy to supervised systems for asymmetric categories, and scalable generalization to unseen categories. The limitations on symmetric objects highlight meaningful open research problems around representation of pose uncertainty and symmetry in self-supervised learning. The geometric bottleneck strategy delineated here is immediately relevant for scalable, category-agnostic vision and robotics systems.

Figure 6: Qualitative predictions across benchmarks. Predicted canonical axes (top) and meshes rendered under predicted poses (bottom), illustrating across-category frame consistency without per-category cues.