Scene-Centric Unsupervised Video Panoptic Segmentation

Abstract: Video panoptic segmentation (VPS) aims to jointly detect, segment, and track all objects while partitioning the video into semantically consistent regions. We introduce the task setting of unsupervised VPS, omitting any human supervision. Existing unsupervised scene understanding works mainly focused on image segmentation tasks; the video domain remains underexplored. We propose VideoCUPS, the first unsupervised VPS approach. VideoCUPS generates temporally consistent panoptic video pseudo-labels from scene-centric videos by exploiting unsupervised depth, motion, and visual cues. Training on these pseudo-labels using a novel Video DropLoss yields an accurate, unsupervised VPS model. To benchmark progress, we introduce a comprehensive evaluation protocol and four competitive baselines, extending state-of-the-art unsupervised panoptic image and instance video segmentation models to VPS. VideoCUPS outperforms all baselines and demonstrates strong label-efficient learning. With VideoCUPS, our evaluation protocol, and baselines, we provide a strong foundation for future research on unsupervised VPS.

• The paper introduces VideoCUPS, the first unsupervised method for scene-centric video panoptic segmentation using motion, depth, and SSL cues.
• The paper details a novel pseudo-label generation process that fuses optical flow, monocular depth, and region-growing instance segmentation to ensure temporal consistency.
• The paper demonstrates robust performance and cross-domain generalization, significantly surpassing unsupervised baselines on standard VPS benchmarks.

Scene-Centric Unsupervised Video Panoptic Segmentation: Technical Summary and Implications

Introduction and Problem Setting

The paper "Scene-Centric Unsupervised Video Panoptic Segmentation" (2606.04925) introduces the first method for unsupervised Video Panoptic Segmentation (VPS), which requires joint semantic/instance segmentation and identity association over time without any reliance on human annotations. The work is motivated by the limitations of supervised approaches that depend on labor-intensive pixel-level annotations, especially for the temporal domain, which is critical but suffering from scalability and quality bottlenecks. Recent progress in self-supervised visual representations and unsupervised segmentation has improved scene understanding for static images, yet comprehensive unsupervised segmentation and tracking for complex, dynamic environments in the video regime remained largely unexplored.

Method: VideoCUPS—Monocular Pseudo-labeling and Training

The core contribution is VideoCUPS, the first unsupervised end-to-end method for scene-centric VPS trained solely from monocular videos. The approach eschews both class labels and stereo views, relying exclusively on unsupervised cues—motion (optical flow), monocular depth, and SSL-based visual features—to generate temporally consistent panoptic pseudo-labels.

The pseudo-label generation stage leverages:

• Unsupervised Optical Flow: SMURF is used for dense motion estimation.
• Monocular Depth and Motion Masking: DynamoDepth provides depth estimates and per-pixel motion probabilities, enabling discrimination between static and dynamic regions.
• Region-Growing Instance Segmentation: Non-rigid instances are extracted via motion-based region growing, thresholded on the motion mask, with iterative merging based on both relative depth and flow similarity.
• Semantic Segmentation: DINO-based SSL features are clustered with $k$-means. High-resolution predictions are fused using a depth-based weighting scheme, with CRF post-processing for spatial consistency.
• Temporal Consistency: Instances and semantics are temporally propagated/associated using optical flow warping and Hungarian matching, and short-lived tracks are filtered.

Figure 1: VideoCUPS pseudo-label generation pipeline combining unsupervised motion, monocular depth, and SSL features for temporally coherent panoptic pseudo-labels.

Figure 2: VideoCUPS yields temporally consistent, dense instance tracks and captures non-rigid motions compared to CUPS-based approaches.

Ultimately, per-frame semantic and instance pseudo-labels are aligned into temporally consistent panoptic pseudo-labels and separated into "thing" and "stuff" categories using distributional statistics over the set of semantic classes.

Training employs these pseudo-labels to optimize a Panoptic Cascade MaskTrack R-CNN (backbone initialized with DINO), including:

• Video DropLoss: Only instance predictions with sufficient pseudo-label overlap are supervised, enabling the model to hypothesize additional (potentially static) objects and tracks not recoverable by motion segmentation.
• Self-enhanced Video Copy-Paste Augmentation: Copy-paste is performed not only with pseudo-labels but also using the model's own confident predictions, improving detection/tracking especially for small or static objects.

Baselines, Evaluation Protocol, and Proposed Metrics

Four strong baselines are constructed by extending state-of-the-art unsupervised instance, semantic, and panoptic segmentation methods to the video setting, including both monocular and stereo approaches. The evaluation protocol introduces extension of STQ (Segmentation and Tracking Quality)—composed of AQ and SQ—to the unsupervised setting by employing pseudo-class alignment (Hungarian matching), while also reporting VPQ and class-wise results.

Results

In-domain Results (Cityscapes-VPS)

VideoCUPS surpasses all unsupervised baselines on Cityscapes-VPS by 4.4–12.3% STQ, even outperforming CUPS+SORT despite being monocular-only (see Table 1 in the paper and qualitative results).

Figure 3: VideoCUPS provides more temporally stable, denser instance tracks and resolves non-rigid or fine-grained motions, especially compared to prior CUPS variants and VideoCutLER-based methods.

Cross-Domain Generalization

Robust performance is observed across KITTI-STEP, Waymo, MOTS, and DAVIS datasets, where VideoCUPS consistently outperforms all baselines. This demonstrates strong label-agnostic generalization across diverse scene distributions and motion regimes.

Figure 4: VideoCUPS demonstrates robust generalization for KITTI-STEP, with improved instance/semantic alignment and track consistency.

Figure 5: VideoCUPS yields strong results on MOTS, segmenting and tracking pedestrians robustly with fine-grained mask boundaries and fewer spurious associations.

Ablations and Analysis

• Pseudo-label Generation: Inclusion of instance propagation, temporal smoothing, and depth-guided semantic inference incrementally improves STQ and AQ. Monocular pseudo-labels recover more dynamic instances (at the cost of slight loss in SQ versus perfect depth from stereo).
• Training Pipeline: Video DropLoss and self-enhanced copy-paste are critical; they significantly close the gap to supervised upper bounds, particularly on track association and small object detection.
• Dynamic vs. Static: The pipeline recovers both moving and some static objects, highlighting generalization beyond motion-based initialization, a consequence of the sparseness-tolerant DropLoss.
• Few-Shot Label Efficiency: VideoCUPS initialization allows fine-tuning on only 10% of annotated data to match the STQ achieved with full labeled training and random initialization. This establishes strong empirical evidence for label-efficient learning scenarios.

Figure 6: Example of robust tracking through partial occlusion; while full occlusions are not recoverable, the system achieves partial, temporally coherent tracks in challenging settings.

Implications and Future Directions

VideoCUPS represents a significant expansion of unsupervised scene understanding by integrating spatial and temporal coherence, non-rigid motion, and high-frequency mask accuracy in a label-free setting. The practical impact is pronounced for deployment in environments where annotation is non-existent or infeasible (robotics, medical imaging, dynamic open-world settings). The method is natively monocular and self-supervised, obviating constraints imposed by stereo hardware or human supervision.

Theoretical Implications

• Motion and Depth as Inductive Priors: The work substantiates the necessity and sufficiency of motion and depth cues (even unsupervised) for constructing high-fidelity temporal panoptic structure in the absence of semantic priors.
• Instance-agnostic Pseudo-labels: The combination of region growing and SSL features in a scene-centric, non-object-centric context enables flexible transfer to out-of-distribution domains.
• DropLoss for Sparsity Tolerance: The loss design allows supervised learning on partial, noisy, and temporally inconsistent labels, encouraging the model to hallucinate plausible instance hypotheses not captured by original cues.

Future Work

• Unsupervised Discovery of Static Objects: Current initialization is motion-centric; integrating MaskCut-like or language-driven object discovery could expand to static but salient entities.
• Post-occlusion Recovery: Improving tracking through occlusion and extending beyond short video clips using long-term temporal association mechanisms.
• Open-Vocabulary and Hierarchical Taxonomies: Future benchmarks should focus on taxonomy induction and open-set evaluation, as unsupervised categories may not align with human-defined classes.
• Depth and Motion Generalization: Advances in unsupervised monocular depth/flow that generalize to arbitrary domains would further enhance pseudo-label fidelity and scalability.

Conclusion

VideoCUPS establishes a technically rigorous, modular framework for unsupervised scene-centric video panoptic segmentation. It closes the gap between supervised and unsupervised video understanding by integrating SSL representations with unsupervised motion and depth, achieves strong generalization, and demonstrates label efficiency for downstream applications. The framework sets a methodological baseline for future research into large-scale, label-free, and generalizable spatiotemporal scene parsing.