VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding

Abstract: Recent advancements in language-model-based video understanding have been progressing at a remarkable pace, spurred by the introduction of LLMs. However, the focus of prior research has been predominantly on devising a projection layer that maps video features to tokens, an approach that is both rudimentary and inefficient. In our study, we introduce a cutting-edge framework, VaQuitA, designed to refine the synergy between video and textual information. At the data level, instead of sampling frames uniformly, we implement a sampling method guided by CLIP-score rankings, which enables a more aligned selection of frames with the given question. At the feature level, we integrate a trainable Video Perceiver alongside a Visual-Query Transformer (abbreviated as VQ-Former), which bolsters the interplay between the input question and the video features. We also discover that incorporating a simple prompt, "Please be critical", into the LLM input can substantially enhance its video comprehension capabilities. Our experimental results indicate that VaQuitA consistently sets a new benchmark for zero-shot video question-answering tasks and is adept at producing high-quality, multi-turn video dialogues with users.

• The paper introduces VaQuitA, a framework that enhances the alignment between video content and text queries using CLIP-score guided frame selection.
• The paper employs a trainable Video Perceiver and Visual-Query Transformer to condense and synchronize video embeddings with textual inputs for improved video QA and dialogue.
• The paper demonstrates that strategic prompt engineering, such as adding 'Please be critical', significantly boosts the LLM's performance in video understanding tasks.

The paper introduces a new framework named VaQuitA designed to enhance the performance of LLMs in the context of video understanding, particularly for video-based question answering and dialogue systems. Video question answering entails understanding the content of a video and answering questions related to it, which is a challenging task as it requires effective alignment and integration of information from the video and the query text.

Unlike previous approaches that predominantly relied on projecting video features directly into token space using a simple projection layer, the authors of this paper developed VaQuitA with three novel components aimed at improving the alignment between the video and textual information.

The first component, Data Alignment, focuses on selecting frames from the video based on their relevance to a given question. This is achieved through a sampling method guided by CLIP-score rankings, abandoning the uniform sampling typically used, which often misses important information. By choosing frames that are more likely related to the question, VaQuitA can provide more contextually relevant features to the LLM.

The second element, Feature Alignment, introduces two mechanisms: a trainable Video Perceiver and a Visual-Query Transformer (VQ-Former). The Video Perceiver condenses video features into a more manageable set of embeddings for the LLM to process. The VQ-Former, meanwhile, ensures that these video feature embeddings are aligned with the textual query, creating a more coherent interplay between the video input and the question being asked.

A surprising and significant finding reported in the paper is the role of Prompt Engineering. The researchers discovered that adding a seemingly simple prompt "Please be critical" before the question greatly improved the LLM’s performance in video understanding tasks. This insight suggests that guiding the model with the right prompt can lead to a more critical and effective analysis by the LLM.

The paper boasts experimental results indicating that VaQuitA achieves state-of-the-art performance on zero-shot video question answering tasks. It also demonstrates the model's capability in facilitating high-quality, multi-turn video dialogues, setting new benchmarks for the given tasks across several datasets.

In summary, VaQuitA marks a significant advancement in aligning video content with textual queries for video question answering. It achieves this through sophisticated frame selection methods and enhancements in feature integration, underscored by the strategic use of linguistic prompts to refine the model’s understanding capabilities.