Tracking Anything with Decoupled Video Segmentation

Abstract: Training data for video segmentation are expensive to annotate. This impedes extensions of end-to-end algorithms to new video segmentation tasks, especially in large-vocabulary settings. To 'track anything' without training on video data for every individual task, we develop a decoupled video segmentation approach (DEVA), composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. Due to this design, we only need an image-level model for the target task (which is cheaper to train) and a universal temporal propagation model which is trained once and generalizes across tasks. To effectively combine these two modules, we use bi-directional propagation for (semi-)online fusion of segmentation hypotheses from different frames to generate a coherent segmentation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks including large-vocabulary video panoptic segmentation, open-world video segmentation, referring video segmentation, and unsupervised video object segmentation. Code is available at: https://hkchengrex.github.io/Tracking-Anything-with-DEVA

Decoupled Video Segmentation for “Tracking Anything”

The paper "Tracking Anything with Decoupled Video Segmentation" addresses an intrinsic problem in video segmentation: the high annotation costs of video training data limit the extensibility of video segmentation algorithms, especially in scenarios with a large number of classes. Instead of relying on extensive video annotation for each distinct task, the authors propose the Decoupled Video Segmentation Approach (DEVA), which leverages both task-specific image-level segmentation and universal temporal propagation models. This decoupled approach avoids over-reliance on target domain video data, promising enhancements in generalization across diverse segmentation tasks.

Key Contributions and Methodology

1. Decoupling Segmentation Tasks: DEVA splits the video segmentation task into image-level segmentation and task-agnostic temporal propagation, enabling efficient training and increased adaptability. By using pre-trained universal promptable models, such as Segment Anything Model (SAM), within the segmentation pipeline, DEVA takes advantage of existing powerful image segmentation capabilities. This hierarchical approach effectively handles variations in segmentation complexity and class volume, a considerable improvement over end-to-end methods.
2. Bi-directional Propagation: Bi-directional temporal propagation plays a critical role in achieving segmentation coherence over time by fusing hypotheses from various frames. This mechanism refines image segmentation results through noise reduction strategies and consolidates new observations to adapt seamlessly to emerging objects or changes within a video.
3. Application on Data-scarce Tasks: DEVA is validated on several challenging video segmentation tasks, namely, large-vocabulary video panoptic segmentation, open-world video segmentation, referring video segmentation, and unsupervised video object segmentation. In data-scarce environments, these tasks benefit from DEVA’s capacity to generalize based on limited annotations by effectively using external task-agnostic temporal information.

Experimental Evaluation and Results

The empirical evaluation demonstrates that DEVA meets or exceeds the performance of state-of-the-art end-to-end video segmentation frameworks, especially in large and open vocabulary settings:

• VIPSeg Dataset: DEVA achieves favorable results in Video Panoptic Quality (VPQ) across multiple experimental setups with varied image models. Its ability to deliver superior performance, particularly in long-range associations, underscores the effectiveness of its bi-directional temporal framework.
• Open-World Video Segmentation: On datasets such as BURST, DEVA demonstrates improved Open World Tracking Accuracy, highlighting its robustness in scenarios without predefined object categories. It efficiently uses task-specific image models, like EntitySeg, illustrating the flexibility and adaptability of its architecture.
• Real-Time Applications: While DEVA may not match the speed of some specialized end-to-end solutions, its performance gain and versatility in handling varied segmentation requirements provide compelling benefits for practical applications. Future developments may focus on speeding up DEVA without compromising accuracy.

Implications and Future Directions

DEVA presents a promising shift from large-scale dependency on video training data by decomposing the video segmentation task and leveraging advances in image segmentation. Practically, this shift means broader accessibility to sophisticated video analysis tools across industries, from autonomous navigation to content creation, without the prohibitive costs currently associated with large-scale video annotation.

Theoretically, DEVA’s success indicates that high-performing video segmentation does not necessitate expansive end-to-end training under all circumstances. Future research may capitalize on the adaptive strengths of decoupled architectures, optimizing models for increasingly nuanced segmentation tasks in real-time applications or further exploring the integration of advanced temporal propagation schemes.

Overall, DEVA is indicative of a broader trend toward modular, versatile AI solutions that efficiently leverage existing models. As such, it sets a benchmark for future advancements in adaptable video segmentation methodologies.