Seek to Segment: Active Perception for Panoramic Referring Segmentation

Abstract: Existing referring segmentation models passively process static images captured from fixed perspectives, limiting their applicability in Embodied AI, where agents must perform active perception in the continuous 360$^\circ$ environments. To bridge this gap, we introduce a novel task: Active Panoramic Referring Segmentation (APRS). In this setting, an agent is required to adjust its viewing direction ($Δθ, Δφ$) to explore the 360$^\circ$ environment, seeking the object specified by a user instruction for segmentation. To tackle this challenging task, we propose PanoSeeker, a memory-augmented agent for efficient APRS. Rather than relying on heuristic scanning, PanoSeeker integrates a Vision-LLM (VLM) with EgoSphere, an explicit spatial visual memory. By progressively integrating sequential local observations into a unified 360$^\circ$ representation, EgoSphere enables the agent to plan efficient and non-redundant search trajectories. Once the target is found, the agent performs active viewpoint alignment and outputs the segmentation mask. Furthermore, we curate an expert-annotated search trajectory dataset with memory timelines for Supervised Fine-Tuning, followed by Reinforcement Learning post-training to explicitly optimize PanoSeeker's exploration efficiency. Extensive experiments on our newly established APRS benchmark demonstrate that PanoSeeker achieves superior search efficiency and segmentation accuracy, significantly outperforming adapted state-of-the-art baselines.

• The paper demonstrates that integrating an explicit EgoSphere memory in PanoSeeker significantly enhances exploration efficiency and segmentation accuracy in 360° scenes.
• It formulates the active panoramic referring segmentation (APRS) task by requiring dynamic viewpoint control and spatial reasoning over continuously evolving scenes.
• Empirical results show PanoSeeker outperforms baselines with a 75.4% success rate, 55.8% mIoU, and improved path efficiency using supervised finetuning and reinforcement learning.

Active Panoramic Referring Segmentation: A Technical Overview of PanoSeeker

Introduction and Motivation

Referring segmentation—identifying and segmenting objects in images conditioned on natural language expressions—has advanced considerably through the adoption of VLMs, but prevailing models operate on static, fixed-view imagery. This passive paradigm is fundamentally misaligned with the requirements of Embodied AI, where agents interactively explore 360-degree environments, localize, and segment referents defined by language. Bridging this gap, "Seek to Segment: Active Perception for Panoramic Referring Segmentation" (2607.02497) formalizes the novel task of Active Panoramic Referring Segmentation (APRS), requiring active viewpoint control and spatial reasoning in a continuous $360^\circ$ setting. To support APRS, the authors introduce the benchmark dataset, a taxonomy of spatial linguistic expressions, and PanoSeeker—the first VLM-based agent equipped with an explicit, geometrically consistent visual memory tailored for efficient panoramic exploration.

Figure 1: APRS requires an agent to search for and segment a language-defined target in a $360^{\circ}$ environment, dynamically adjusting viewing direction to localize the referent.

Task Formulation and Benchmark Characteristics

APRS targets complex real-world scenarios where the referent is initially out of view, necessitating active exploration. The agent incrementally samples the $360^\circ$ scene through perspective crops parameterized by azimuth ($\theta$) and elevation ($\phi$) and updates its memory after each observation. Task success is jointly determined by (1) spatial proximity of the agent’s final viewpoint to the target (geodesic threshold), (2) the segmentation mask’s IoU (≥0.5), and (3) the efficiency of the search trajectory, as evaluated by weighted path-length-based metrics (SPL) and step count.

The APRS benchmark is constructed from 4,971 high-resolution panoramic scenes (drawn from 360-Indoor, PANDORA, SUN360), featuring 7,420 unique episodes balanced over a taxonomy of spatial instructions:

• Egocentric (EGO): Directional cues relative to agent’s view (e.g., “Turn right to find the sofa”)
• Unique-Attribute (UNIQ): Refers using fine-grained properties (e.g., color, shape)
• Allocentric (ALLO): Anchor-based spatial relations (e.g., “the chair opposite the sofa”)
• Multi-hop (MULTIHOP): Integrates viewpoint change and complex spatial cues

The dataset annotation pipeline combines human expertise in constructing active search trajectories and spatial expressions with model-based segmentation and semantic consistency checks, enabling a high-fidelity, large-scale resource for APRS.

The PanoSeeker Framework

PanoSeeker addresses core deficiencies of traditional log-based agent memory by introducing EgoSphere, a spatially explicit and geometrically structured memory system. EgoSphere differs critically from prior agent memories by providing a projection-consistent, continuously updated ERP canvas, which integrates sequential observations onto a unified panoramic map. This persistent memory resolves key issues: redundant exploration, lack of global spatial context, and geometric distortion arising from naive concatenation or text-based logging.

The PanoSeeker pipeline is as follows:

• At each timestep, the agent observes $V_t$ (current view), updates EgoSphere memory $M_t$, and conditions on the user instruction $I$.
• By ingesting both the current observation and the globally consistent memory canvas, PanoSeeker (VLM backbone: Qwen3-VL) predicts the optimal next action—a movement $(\Delta\theta_t, \Delta\phi_t)$ or STOP.
• Upon indication of target localization, active alignment is conducted to refine the viewpoint and produce a segmentation mask exploiting modern foundation models (SAM-3).

  Figure 2: PanoSeeker architecture: VLM + EgoSphere memory predicts camera controls and performs segmentation after viewpoint alignment. Exploration policy is RL-optimized for efficiency.

Memory Augmentation: EgoSphere

EgoSphere projects each new FoV’s pixels onto a $360^\circ$ ERP map, explicitly marking both visited and unexplored regions. Memory augmentation also includes:

• Crosshair overlays for egocentric/allocentric alignment
• Latitude-longitude grids for angular reasoning
• Trajectory visualization to avoid redundant revisitation

As evidenced in the ablation studies, replacing sequence-based log or buffer memory with EgoSphere yields a substantial increase in both path efficiency and task success metrics.

Training Paradigm: SFT + RL

Initial supervised finetuning on expert-annotated trajectories imparts imitation capabilities, while subsequent reinforcement learning (GRPO) explicitly optimizes for exploration efficiency (rewarding direct paths and penalizing dead-ends). The composite policy maximizes both efficiency and segmentation quality.

Empirical Analysis

PanoSeeker outperforms all baselines, including:

• Static panoramic segmentation methods (single-pass ERP processing): Severely limited in dealing with geometric distortions and incapable of reasoning about spatial instructions.
• Heuristic scanning: Covers full $360^{\circ}$0, but at the cost of excessive path length and minimal reasoning.
• VLM-based agents with text log memory: Suffer from low global spatial coherence and inefficient search trajectories.

Notably, PanoSeeker achieves a 75.4% success rate, 55.8% mIoU, and a 0.57 SPL with only 4.8 steps per episode, representing a significant leap in both accuracy and efficiency compared to the strongest static and active baselines.

In ablations, each stage—EgoSphere integration, supervised trajectory training, and GRPO-based RL—contributes incrementally to the final performance, with the explicit geometric memory providing the most substantial boost to exploration efficiency and success rate.

Figure 3: Visualization of search trajectories and ERP memory accumulation, with PanoSeeker efficiently localizing targets and minimizing redundant exploration.

Implications and Future Directions

The introduction of APRS and the EgoSphere-based agent architecture has broad implications for active perception and Embodied AI research:

• Methodological: EgoSphere demonstrates that explicit geometric memory representations are critical for spatially structured active perception tasks, outperforming context-augmented and buffer-based memory designs.
• Practical: PanoSeeker’s sample-efficient exploration and high segmentation fidelity provide a scalable solution for language-guided robotic manipulation, assistive real-world search, and interactive navigation in panoramic or VR/AR environments.
• Benchmarking: The APRS benchmark constitutes a new standard for comprehensive evaluation of spatially-aware VLM agents in non-static, language-grounded settings.

Potential avenues for extension include integrating differentiable mapping with depth/3D cues, scaling memory systems for lifelong open-world exploration, and enabling continual language-instruction adaptation over long-horizon agent deployments. Further research should examine synergistic learning of perception, memory, and control through unified architectures, and address challenges such as ambiguity, occlusion, and real-time constraint adherence in dynamic environments.

Conclusion

This work establishes APRS as a rigorous testbed for language-guided, active visual perception, supported by a robust dataset and well-defined task protocol. PanoSeeker’s EgoSphere memory and dual-phase learning paradigm yield state-of-the-art performance, substantiating the necessity of explicit, spatially structured visual memory for efficient panoramic object localization and segmentation in Embodied AI. The benchmark and framework are poised to stimulate subsequent research on spatial reasoning, efficient exploration, and robust segmentation under challenging, real-world agent settings.