First Place Solution to the Multiple-choice Video QA Track of The Second Perception Test Challenge

Abstract: In this report, we present our first-place solution to the Multiple-choice Video Question Answering (QA) track of The Second Perception Test Challenge. This competition posed a complex video understanding task, requiring models to accurately comprehend and answer questions about video content. To address this challenge, we leveraged the powerful QwenVL2 (7B) model and fine-tune it on the provided training set. Additionally, we employed model ensemble strategies and Test Time Augmentation to boost performance. Through continuous optimization, our approach achieved a Top-1 Accuracy of 0.7647 on the leaderboard.

• The paper demonstrates a winning solution by fine-tuning QwenVL2 with high-resolution instruction tuning, achieving a top-1 accuracy of 76.47%.
• It employs ensemble strategies and Test Time Augmentation to mitigate biases and enhance the performance of multiple models.
• The approach leverages cross-validation and low-rank adaptation for efficient handling of video data, paving the way for scalable multimodal comprehension.

Overview of the First Place Solution to the Multiple-choice Video QA Track of The Second Perception Test Challenge

This paper presents an in-depth analysis of the winning solution in the Multiple-choice Video Question Answering (Video QA) track of The Second Perception Test Challenge. The research leverages the QwenVL2 (7B) model, a leading-edge multimodal video understanding framework, fine-tuned extensively to meet the challenge's demands. The authors employed innovative approaches, including model ensemble strategies, Test Time Augmentation (TTA), and cross-validation methods, achieving a Top-1 accuracy of 0.7647 on the challenge leaderboard.

Methodological Approach

The solution primarily hinges on the use of the QwenVL2 (7B) model for its robust baseline capabilities, as initially demonstrated by its zero-shot performance marked by a 0.61 accuracy score. Advancing further, the team incorporated High-Resolution Instruction Tuning (HR-IT) tailored to the dataset's high-resolution video content. Notably, this phase included a 5-fold cross-validation to enhance model robustness. The application of Low-Rank Adaptation (LoRA) facilitated efficient parameter handling without compromising performance, leveraging hardware resources effectively with four NVIDIA A6000 GPUs.

Model Performance Enhancements

Several key enhancements were pivotal:

1. Model Ensemble Strategy: Six models were trained, and their predictions aggregated using majority voting, which elevated the accuracy to 0.7551.
2. Test Time Augmentation (TTA): This technique addressed the positional bias by shuffling multiple-choice options, conducting inference across different permutations, and implementing majority voting.
3. High-Resolution Inference: By experimenting with more frames and greater resolution, noticeable gains in comprehension were achieved, particularly with 60 frames and a resolution of 560 x 630 pixels.

Ensemble Strategy and Numerical Results

The finalized ensemble strategy consisted of 31 sets of inference results, which were weighted distinctively to maximize model performance across varying conditions. Its deployment necessitated intricate balancing of inference outputs drawn from baseline, HR-IT, and TTA-influenced models, ultimately achieving the reported top accuracy.

Implications and Future Prospects

The findings from this research hold significant implications:

• Practical Applications: The results demonstrate scalable model configurations that are applicable to broader challenges in multimodal video comprehension, highlighting the importance of advanced ensemble techniques in achieving robust performance.
• Theoretical Insights: This work contributes to the theoretical understanding of how high-resolution video processing and model architectures like QwenVL2 can handle temporal and spatial complexities inherent in video QA.

Looking forward, future developments in AI may expand upon these methodologies by further refining model architectures and exploring the integration of even more diverse datasets to enhance generalizability. Exploring adaptive learning rates and novel ensemble learning strategies may also yield additional performance gains. The paper's approach paves the way for subsequent advancements in video-based AI systems, promising enhanced interpretability and accuracy.