Panoramic Scene Analysis: A Survey from Distortion-Aware Engineering to Sphere-Native Foundation Modeling

Abstract: Panoramic images capture the complete visual sphere in a single frame, providing spatial context unattainable by conventional cameras. Yet this completeness comes at a geometric cost: the 2-sphere cannot be faithfully mapped to the plane, and every planar representation introduces distortions that violate the assumptions underlying standard vision architectures. This survey traces the evolution of panoramic scene analysis along a methodological trajectory, from projection-based adaptation, through distortion-aware engineering, to sphere-native modeling and geometry-aware tokenization for foundation models, and argues that this evolution reflects a progressive deepening of geometric commitment rather than a simple accumulation of techniques. We organize the literature along two orthogonal dimensions: architectural design (how operators interact with spherical geometry) and training paradigm (how knowledge is transferred across domains). Covering dense prediction (semantic segmentation, depth estimation, and room layout estimation), unified multi-task understanding, open-world perception, vision-language reasoning, and dynamic video analysis, we identify a central unresolved tension: among the methods surveyed, none simultaneously delivers strict spherical equivariance and full reuse of perspective-pretrained foundation-model weights, and we argue that this is a structural rather than incidental gap. We further expose five systematic gaps in current evaluation protocols, namely the absence of spherical-area-weighted metrics, seam-consistency testing, polar-robustness stratification, cross-projection generalization, and open-world protocol standardization, and propose a six-point research roadmap toward general-purpose panoramic intelligence. The corresponding repository is publicly available at: https://github.com/zhuqinfeng1999/Awesome-Panoramic-Scene-Analysis.

• The paper introduces a taxonomy for panoramic scene analysis, contrasting projection-based, distortion-aware, and sphere-native methods to address intrinsic geometric challenges.
• It demonstrates that effective geometry-aware tokenization and interface adaptation are crucial for leveraging perspective foundation models on 360° data.
• The study highlights unresolved issues in evaluation metrics and dynamic vision, underscoring the need for unified, scale-trained spherical models.

Panoramic Scene Analysis: From Distortion-Aware Engineering to Sphere-Native Foundation Modeling

Introduction and Theoretical Context

Panoramic imaging has become integral for tasks demanding holistic scene understanding, including AR/VR, embodied AI, and autonomous systems. Unlike perspective imagery, panoramas are signals on the 2-sphere ($\mathbb{S}^2$), whose positive curvature and SO(3) isometry group introduce fundamental geometric challenges for downstream machine learning. The impossibility of an isometric mapping from $\mathbb{S}^2$ to $\mathbb{R}^2$ mandates trade-offs between spatial continuity, area/angle preservation, and architectural compatibility with standard vision models. All major planar projections induce distortions that propagate inductive biases and failure modes in conventional models.

Figure 1: The panoramic imaging pipeline highlights how diverse capture systems project signals onto the unit sphere, necessitating subsequent planar projection that embeds geometric biases into all downstream learning systems.

Any adaptation of contemporary vision architectures must explicitly address these distortions. Panoramic scene analysis is thus a test-bed for the geometric depth of vision architectures, and progress in this domain maps the trajectory by which vision research has moved from heuristic distortion-correction to intrinsic spherical geometry modeling.

Methodological Taxonomy: Architecture and Training Paradigms

The survey formalizes the field using a two-dimensional taxonomy: (1) architectural geometric fidelity from projection-based adaptation (PB), distortion-aware engineering (DA), to sphere-native modeling (SN), and geometry-aware foundation-model interfaces (GT); (2) training and transfer paradigms, ranging from supervised learning on scarce panoramic data, through unsupervised domain adaptation (UDA), to leveraging perspective foundation models. The orthogonality of these axes is crucial: only recently do methods achieve innovation on both, and the upper-right quadrant (sphere-native $\times$ foundation-model-scale transfer) remains underexplored.

Figure 2: Two-dimensional taxonomy showing the joint landscape of geometric architectural awareness and training/transfer paradigm sophistication; sphere-native foundation model transfer is an open frontier.

Architectural trends:

• Projection-based Adaptation (PB): Disassemble panoramas into ERP, cubemaps, or patches, enabling direct use of planar models (e.g., BiFuse, Tangent, CViT). This process minimizes architectural change but introduces seams and locality artifacts.
• Distortion-Aware Engineering (DA): Inject analytic compensation for known ERP/cubemap distortions at the level of sampling grid deformation (SphereNet), kernel adaptation (KTN), or attention mechanisms (Trans4PASS, SGAT4PASS). However, these are projection-specific and typically tied to ERP.
• Sphere-Native Modeling (SN): Define computation intrinsically on $\mathbb{S}^2$ (e.g., Spherical CNNs, spectral convolution, icosahedral mesh-based methods, gauge-equivariant layers). While enabling exact rotation equivariance and geometric consistency, these architectures are computationally expensive and incompatible with planar pretraining.
• Geometry-Aware Tokenization/GT: Engineer the input/output interface to make foundation models robust to spherical geometry without altering core weights (e.g., ERP-RoPE, SDPE, sequence reformation for SAM2, kernel-level compatibility adaptors).

Parallel advances in domain adaptation and foundation-model transfer have allowed methodologies to decouple supervisory signal origin and model design, facilitating Pin2Pan transfer, UDA, SFUDA, and pseudo-label distillation using large-scale perspective-pretrained models.

Task-Specific Advances: Dense Prediction, Unified Understanding, Foundation-Model Adaptation

Progress in panoramic dense prediction spans segmentation, depth estimation, and room layout. Early systems (e.g., PASS) showed that direct deployment of perspective-trained architectures leads to severe accuracy degradation ($\sim$50% mIoU drop). This has driven a move first to DA transformer architectures (Trans4PASS, DATR) and then to geometric priors and unified multi-task models (SGFormer, HUSH).

Current foundation-model adaptation approaches demonstrate that geometric-aware interface adaptation can efficiently re-use SAM- and DINOv2-scale pretrained representations for 360$^\circ$ data (OmniSAM, SAP) but rely heavily on sophisticated patch/trajectory encoding and custom protocols. For panoramic depth, construction approaches (SphereViT, UniK3D) using large panoramic corpora and custom geometric embeddings are yielding further margin over vanilla adaptation.

Panoramic open-world understanding introduces unique challenges—novel-class appearance is entangled with distortion artifacts and seam discontinuities, and the benefits of $360^\circ$ spatial context are underexploited. Existing open-world segmentation and OOD detection systems (OPS, PanOoS) remain preliminary in evaluation and scope.

Foundational Gaps in Evaluation and Datasets

Despite technical advances, evaluation standards do not isolate or probe true spherical understanding. Standard metrics (mIoU, RMSE) over-count polar regions due to ERP oversampling. Seam continuity, latitude stratification, and projection-agnostic generalization are almost never reported, creating conditions for superficial metric gains. No panoramic foundation model has yet been trained at scale to parity with DINOv2/CLIP using native spherical data, and high-quality annotated data for outdoor and temporal domains lags the indoor static setting by one to two orders of magnitude.

Vision-LLMs and Panoramic Spatial Reasoning

The application of multimodal LLMs and vision-LLMs (VLMs) to panoramas exposes new architectural vulnerabilities. Off-the-shelf VLMs trained on perspective imagery—despite encoder/decoder capacity—fail to maintain spatial intelligence under ERP distortions, seam effects, and non-uniform patch spatial semantics. Geometry-aware patch tokenization and encoding (ERP-RoPE, sparse attention) yield significant empirical improvement but do not close the gap on reasoning tasks requiring global geodesic awareness, 3D localization, or multimodal action.

Dynamic Panoramic Perception and Embodied Video

Dynamic (video) panoramic perception remains at an early phase compared to its static counterpart. Extensions of video object tracking (SOT/MOT/VOS), flow, and temporal segmentation encounter unique challenges: pixel motion varies non-linearly with latitude, objects naturally wrap around seams, and persistent visibility breaks standard temporal association heuristics.

Figure 3: Four fundamental challenges of dynamic panoramic analysis: distortion-varying motion, seam wrap-around, continuous visibility, and global ego-motion shifts.

State-of-the-art methods (360VOT, OmniTrack, PanoVOS) resolve some issues via distortion-aware region representations, gnomonic/bounding-FoV boxes, and seam integration, but there is little evidence of temporally sphere-native computation; all memory and association modules are planar.

Synthesis and Outlook: Open Challenges and Roadmap

Key conclusions and future directions include:

1. Architecture/Pretraining Trade-Off: There is an unresolved trade-off between exact rotation equivariance (sphere-native models) and compatibility with foundation-model weights. All current large-scale models prioritize compatibility, settling for engineering spherical awareness at the interface.
2. Tokenization and Representation: No comprehensive comparison exists of the rapidly proliferating geometry-aware tokenization strategies. Unified benchmarking and ablation are needed.
3. Unified Foundation Models: Panoramic models supporting dense, multimodal, and temporal tasks in a unified, sphere-native, transformer-scale architecture remain undeveloped.
4. Evaluation Reform: Protocols remain insufficient; adoption of spherically-weighted metrics, seam/polar stratification, and cross-projection testing is critical for meaningful progress.
5. Data Infrastructure: Automated annotation, synthetic-to-real transfer learning, and projection-agnostic datasets are underexploited for panoramic evaluation.
6. Dynamic/Temporal Panoramic Vision: True time-domain spherical modeling—extending equivariant techniques into recurrent/temporal domains—remains an untouched research area.

Conclusion

The surveyed trajectory from distortion-aware engineering to sphere-native foundation modeling reflects the increasing geometric sophistication required for panoramic intelligence. While practical gains can be achieved through geometric interface engineering and data-driven adaptation of perspective foundations, the theoretical promise of omnidirectional vision will require unified, sphere-native architectures jointly pretrained at scale, with evaluation standards reflecting intrinsic geometric correctness and generalization. A coherent roadmap is laid out for panoramic intelligence: co-design of geometry-aware tokenization, cross-projection-robust backbones, unified multimodal/temporal training, and standardized evaluation will underpin the next generation of $360^\circ$ scene understanding and reasoning models.

---

For complete bibliographic references and more detailed method comparisons, direct consultation of the full survey "Panoramic Scene Analysis: A Survey from Distortion-Aware Engineering to Sphere-Native Foundation Modeling" (2606.27745) and its curated repository is essential.