One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos

Abstract: We introduce VideoLISA, a video-based multimodal LLM designed to tackle the problem of language-instructed reasoning segmentation in videos. Leveraging the reasoning capabilities and world knowledge of LLMs, and augmented by the Segment Anything Model, VideoLISA generates temporally consistent segmentation masks in videos based on language instructions. Existing image-based methods, such as LISA, struggle with video tasks due to the additional temporal dimension, which requires temporal dynamic understanding and consistent segmentation across frames. VideoLISA addresses these challenges by integrating a Sparse Dense Sampling strategy into the video-LLM, which balances temporal context and spatial detail within computational constraints. Additionally, we propose a One-Token-Seg-All approach using a specially designed <TRK> token, enabling the model to segment and track objects across multiple frames. Extensive evaluations on diverse benchmarks, including our newly introduced ReasonVOS benchmark, demonstrate VideoLISA's superior performance in video object segmentation tasks involving complex reasoning, temporal understanding, and object tracking. While optimized for videos, VideoLISA also shows promising generalization to image segmentation, revealing its potential as a unified foundation model for language-instructed object segmentation. Code and model will be available at: https://github.com/showlab/VideoLISA.

• The paper introduces VideoLISA, an MLLM that enables language-instructed reasoning segmentation in videos by effectively integrating temporal dynamics.
• VideoLISA achieves state-of-the-art results on multiple video segmentation benchmarks, including RVOS, MeViS, and the new ReasonVOS.
• VideoLISA has practical applications in surveillance and autonomous driving, advancing multimodal models for complex vision tasks.

Language Instructed Reasoning Segmentation in Videos with VideoLISA

The paper presents VideoLISA, an innovative approach to language-instructed reasoning segmentation in videos, aiming to enhance video object segmentation (VOS) with complex reasoning and temporal understanding capabilities. VideoLISA is a multimodal LLM (MLLM) that extends the capabilities of LISA, a model initially focused on image-based tasks, to dynamic video scenarios. This advancement addresses the significant challenge of integrating temporal dynamics with spatial segmentation in video content, a limitation in many existing methods.

The primary contributions of VideoLISA are twofold. Firstly, it introduces a Sparse Dense Sampling strategy within the video-LLM framework, which optimally balances between temporal context and spatial detail under computational constraints. This technique effectively captures and processes the temporal redundancy inherent in video data, enhancing the model's ability to construct a coherent spatiotemporal narrative required for precise segmentation tasks. Secondly, the One-Token-Seg-All approach is proposed, wherein a specialized <TRK> token is utilized to segment and track objects across multiple video frames, ensuring temporal consistency in the segmentation masks.

VideoLISA leverages the Segment Anything Model (SAM) for mask generation, which, in conjunction with the reasoning capabilities of LLMs, facilitates generating temporally consistent segmentation masks based on diverse language instructions. The extensive evaluations on benchmarks, including the newly introduced ReasonVOS benchmark, underscore VideoLISA's superior performance. This model demonstrates robust capabilities in video object segmentation tasks that necessitate complex reasoning, temporal understanding, and object tracking. Notably, while optimized for videos, VideoLISA exhibits promising generalizability to image segmentation, suggesting its potential as a unified foundation model for language-instructed object segmentation.

The numerical results as presented indicate that VideoLISA achieves competitive or superior performance compared to traditional and LLM-based methods with reasoning capabilities. On standard Referring Video Object Segmentation (RVOS) benchmarks such as Ref-Youtube-VOS and Ref-DAVIS-17, VideoLISA demonstrates state-of-the-art performance, particularly when combined with post-optimization techniques. Additionally, it excels in motion-guided VOS tasks on the MeViS benchmark and reasoning-based segmentation on the ReasonVOS benchmark.

The implications of this research are substantial. Practically, VideoLISA could be employed in various applications such as surveillance, autonomous driving, and interactive video editing, where understanding and segmenting objects in dynamic scenes based on natural language instructions is essential. Theoretically, VideoLISA contributes to the growing body of work integrating LLMs with computer vision tasks, highlighting the potential of multimodal models to tackle complex, real-world challenges.

Future developments in AI leveraging the capabilities of models like VideoLISA could include more refined and efficient integration of temporal information and semantic reasoning across multiple domains. Further exploration into optimizing the computational demands of such models would enhance their applicability and efficiency in real-time applications.

Overall, VideoLISA represents a significant advancement in the domain of language-instructed video segmentation, with both theoretical contributions and practical applications that pave the way for more intelligent, responsive systems capable of understanding and interacting with dynamic real-world contexts.