Fine-tuned CLIP Models are Efficient Video Learners

Abstract: Large-scale multi-modal training with image-text pairs imparts strong generalization to CLIP model. Since training on a similar scale for videos is infeasible, recent approaches focus on the effective transfer of image-based CLIP to the video domain. In this pursuit, new parametric modules are added to learn temporal information and inter-frame relationships which require meticulous design efforts. Furthermore, when the resulting models are learned on videos, they tend to overfit on the given task distribution and lack in generalization aspect. This begs the following question: How to effectively transfer image-level CLIP representations to videos? In this work, we show that a simple Video Fine-tuned CLIP (ViFi-CLIP) baseline is generally sufficient to bridge the domain gap from images to videos. Our qualitative analysis illustrates that the frame-level processing from CLIP image-encoder followed by feature pooling and similarity matching with corresponding text embeddings helps in implicitly modeling the temporal cues within ViFi-CLIP. Such fine-tuning helps the model to focus on scene dynamics, moving objects and inter-object relationships. For low-data regimes where full fine-tuning is not viable, we propose a `bridge and prompt' approach that first uses fine-tuning to bridge the domain gap and then learns prompts on language and vision side to adapt CLIP representations. We extensively evaluate this simple yet strong baseline on zero-shot, base-to-novel generalization, few-shot and fully supervised settings across five video benchmarks. Our code is available at https://github.com/muzairkhattak/ViFi-CLIP.

• The paper demonstrates that fine-tuning CLIP can efficiently adapt image-text models for video understanding, achieving state-of-the-art cross-dataset performance.
• ViFi-CLIP leverages embedding-level fusion and cosine similarity loss to implicitly capture temporal dynamics while reducing computational overhead.
• Experimental results in zero-shot and few-shot settings highlight the model’s robustness and its potential scalability for practical video learning applications.

Fine-Tuned CLIP Models as Efficient Video Learners

The paper "Fine-tuned CLIP Models are Efficient Video Learners" (arXiv ID: (2212.03640)) explores exploiting CLIP's pre-trained image-text embeddings and adapting them to video tasks. The researchers introduce ViFi-CLIP, a simplified approach to fine-tune CLIP for video understanding without relying on extensive architectural changes. This essay provides a comprehensive breakdown of the methodology, experimental settings, and effectiveness of the approach, along with insights into latent capabilities unearthed in CLIP through this adaptation.

ViFi-CLIP: Methodology and Design

ViFi-CLIP is grounded on the pivotal aspect of fine-tuning the already robust CLIP model to bridge the modality gap between static images and dynamic video sequences effectively. Instead of relying on sophisticated architectural-modifications (e.g., self-attention layers), the approach incorporates full fine-tuning of both CLIP’s image and text encoders within the video domain. This enables implicit modeling of temporal cues without the need for additional temporal processing components.

• Embedding-Level Fusion: Frames from videos are processed individually to capture a batch mode similar to images, pooled to form a holistic understanding of action sequences, leveraging the strength of CLIP's learned embeddings.
• Loss Function: The cosine similarity between video embeddings and their corresponding text embeddings is maximized, maintaining a focus on cross-entropy optimization to align multi-modal representations closely.

  Figure 1: Overview of our simple baseline ViFi-CLIP for adapting CLIP~\cite{clip} to videos.

Experimental Evaluation

The study thoroughly evaluates ViFi-CLIP across diverse settings—zero-shot cross-dataset evaluations, base-to-novel class generalization tests, few-shot learning experiments, and fully-supervised settings—demonstrating a consistent level of adaptability and performance gains compared to prior domain-specific models.

• Zero-Shot Setting: Showcases ViFi-CLIP's superior cross-dataset generalization potential when trained on Kinetics-400 and tested on datasets such as HMDB-51 and UCF-101, achieving significant improvements over state-of-the-art models with notable performance leaps in novel datasets (Figure 2).

  Figure 2: t-SNE visualizations for Kinetics-600. For K600, we show the t-SNE visualizations for 160 classes that are non-overlapping with Kinetics-400.

• Few-Shot Learning: Demonstrates robustness and adaptability under limited data conditions, outperforming complex models by better capturing essential spatiotemporal patterns crucial for actions recognition.

Generalization and Robustness

Attention map visualizations (Figure 3) reveal ViFi-CLIP's aptitude in focusing on critical interactions and dynamics, such as object manipulations and participant-object interactions, which pure image-based CLIP models overlook. ViFi-CLIP's enhancements allow it to discern action sequences effectively, focusing on dynamic elements and object relationships crucial for contextual comprehension in videos.

Figure 3: Attention map visualizations of ViFi-CLIP highlight its focus on moving parts in "hammering" and "frisbee catch" categories, emphasizing interactions.

Computational Efficiency

The embedding fusion approach utilized by ViFi-CLIP also introduces efficiency advantages. As indicated, ViFi-CLIP's simplicity results in a lower computational overhead regarding FLOPs and parameters compared to models with explicit temporal modeling capabilities, emphasizing a leverage over conventional architectures by maintaining sparsity in IoU and improving throughput.

Conclusion

This paper establishes that CLIP, when fine-tuned properly, possesses the inherent capacity to efficiently understand video content, outperforming more complex architectures designed with heavy temporal components. Through ViFi-CLIP, the potential of using pre-trained image models in video tasks is expanded subtly yet significantly, heralding opportunities for future advances with CLIP and similar models in video understanding contexts. This study's "bridge and prompt" strategy further ensures potential scalability, opening vistas for low-data regime applications. By leveraging the unified embedding design and tweaking only as necessary, ViFi-CLIP sets a new baseline for practical video learning.