RankVideo: Reasoning-Based Video Reranking

• RankVideo is a reasoning-based text-to-video retrieval framework that reorders candidate videos using raw visual inputs and multimodal reasoning.
• It employs a two-stage retrieval process, combining fast initial matching with transformer-based reranking and a dedicated curriculum for perception-grounded tuning.
• Evaluations on benchmarks like MultiVENT 2.0 demonstrate state-of-the-art improvements in metrics such as nDCG and recall, underscoring its practical impact.

RANKVIDEO denotes a reasoning-based reranking framework developed for text-to-video retrieval. It addresses the challenge of effectively ranking candidate videos by directly leveraging video content and large-scale multimodal reasoning, rather than relying solely on visual-language or text-only matching. RankVideo combines a dedicated curriculum for perception-grounded supervised tuning and advanced reranking losses, integrates efficient data synthesis, and delivers state-of-the-art improvements on benchmarks such as MultiVENT 2.0. The following sections describe its core principles, architecture, loss functions, evaluation, and significance within the broader context of video retrieval and ranking research (Skow et al., 2 Feb 2026).

1. Two-Stage Text-to-Video Retrieval Pipeline

RankVideo operates in a two-stage retrieval and reranking framework:

• First-stage retrieval: An initial retriever (e.g., OmniEmbed, MMMORRF, CLIP, LanguageBind, Video-ColBERT) retrieves the top-K candidate videos for a given natural language query using fast, lightweight matching over video-level or frame-level representations.
• Second-stage reranking: RankVideo reranks these top-K candidates by explicitly reasoning over each query–video pair. Input formats consist of raw RGB frames (up to 32 frames at 2 FPS) for each candidate video and tokenized user queries. The model does not require captions, transcripts, or OCR at inference: all scoring is directly over visual input.

The reranker uses a transformer-based vision-language foundation model (Qwen3-VL-8B-Instruct) and applies a structured prompt per candidate:

“Query: $q$. Is the video relevant to the query? Respond with <answer>yes</answer> or <answer>no</answer>.”

The model extracts the logit margin between the “yes” and “no” tokens to form a relevance score:

$$s_{\theta}(q,v)=\ell_{\theta}(\text{yes}\mid q,v)-\ell_{\theta}(\text{no}\mid q,v)$$

where $\ell_{\theta}(t\mid q,v)$ is the predicted logit for token $t$ (Skow et al., 2 Feb 2026).

2. Training Curriculum and Data Synthesis

RankVideo is optimized in two stages, reflecting distinct objectives:

Stage 1: Perception-Grounded Supervised Fine-Tuning

• Purpose: To ensure the model attains robust, content-grounded video representation before exposure to ranking supervision.
• Data: For each video, a single teacher-generated caption is used. The supervised objective is standard next-token log-likelihood over these captions:

$$\mathcal{L}_{\mathrm{cap}} = - \sum_{t=1}^L \log p_\theta(c_t^{(T)}\mid c_{<t}^{(T)},\,v)$$

Stage 2: Reranking Fine-Tuning

• Data Synthesis:
  • Human queries (1,361) and 7,906 synthetic, reasoning-oriented queries are pooled.
  • Synthetic queries are generated via:
  • Video captioning (Qwen3-Omni-30B)
  • Automated speech transcription (Whisper-Large-v2)
  • OCR over video frames
  • Query/answer pairs filtered by a text-only reasoning teacher (Qwen3-32B) to maximize reasoning complexity and coverage.
  • For each query–video candidate, the reasoning teacher assigns binary labels and a logit margin for “trusted negatives,” “suspected positives” (filtered out), and “ambiguous negatives.”
• Hard negative mining is enforced using explicit logit-margin thresholds (e.g., $\alpha_1=-6$, $\alpha_2=-8$) over teacher scores to ensure difficult and discriminative training batches.

3. Reranking Loss Functions and Optimization

The stage 2 reranking objective is a weighted sum of three loss components, each calibrated by temperature or confidence scaling:

1. Pairwise ranking loss:

   $$p_i = \frac{\exp(s_i/\tau_{pair})}{\sum_j \exp(s_j/\tau_{pair})}, \qquad \tau_{pair}=10$$

   $\mathcal{L}_{\mathrm{pair}} = -\log p_{+}$

2. Teacher confidence distillation (soft BCE with temperature $\tau_{teacher}$):

   $$\mathcal{L}_{\mathrm{T}} = \mathrm{BCE}\left(\mathrm{sigmoid}\left(\frac{s_\theta(q,v)}{\tau_{\mathrm{teacher}}}\right),\, p^{(T)}_{\mathrm{yes}}(q,v)\right)$$

3. Pointwise calibration loss (with softened target for negatives and class weights):

   $$\mathcal{L}_{\mathrm{pt}} = w\,\mathrm{BCE}\left(\mathrm{sigmoid}\left(\frac{s_\theta(q,v)}{\tau_{\mathrm{point}}}\right),\, \tilde{y}\right)$$

The overall loss is:

$$\mathcal{L} = \mathcal{L}_{\mathrm{pair}} + \lambda_{\mathrm{teacher}} \mathcal{L}_{\mathrm{T}} + \lambda_{\mathrm{pt}} \mathcal{L}_{\mathrm{pt}}, \quad \lambda_{\mathrm{teacher}}=5,\,\lambda_{\mathrm{pt}}=0.5$$

Optimization is performed using AdamW, batch sizes (16 in stage 1, 3 in stage 2), and cosine learning rate decay (Skow et al., 2 Feb 2026).

4. Quantitative Performance and Ablation

Evaluation is performed on the MultiVENT 2.0 test set (109,800 videos, multilingual, event-centric queries). Metrics include Recall@N and nDCG@N.

A representative excerpt of test set results:

Method	R@10	nDCG@10	Δ nDCG@10
OmniEmbed	0.523	0.495	–
ReasonRank	0.570	0.543	+9.70%
QVL-I	0.508	0.478	–3.43%
QVL-T	0.515	0.483	–2.42%
RV-1	0.582	0.559	+12.93%
RV-2	0.590	0.566	+14.34%

RankVideo (Stage 2) delivers a +31% average improvement in nDCG@10 versus the first-stage retriever, with the largest uplifts seen when reranking strong but purely visual baselines (e.g. +56.2% for CLIP). Ablations confirm that omitting either the teacher distillation or pointwise loss components causes 1.0–2.0 nDCG point degradation (Skow et al., 2 Feb 2026).

Latency analysis demonstrates that RankVideo’s single-pass logit-margin scoring is only marginally slower than traditional text-based ReasonRank (≈1.02 s vs. ≈0.87 s per (q,v)), while being over 3× faster than chain-of-thought variants (QVL-T). Runtime efficiency is achieved by avoiding chain-of-thought decoding and optimizing only for the answer-logit margin.

5. Methodological Distinctions and Future Directions

Distinctive aspects of RankVideo in the contemporary reranking landscape:

• Video-Native Reasoning: Unlike methods dependent on offline captioning or transcription, RankVideo’s scoring is grounded directly in raw frames, avoiding reliance on intermediate representations at inference.
• Unified Loss: Combining pairwise, pointwise, and teacher distillation objectives enables robust calibration, reduced class imbalance, and improved discriminative power on reasoning-intensive pairs.
• Curriculum Tuning: Initial supervised caption-tuning ensures semantic perception before ranking optimization, maximizing downstream sample efficiency.
• Data Synthesis: Balanced real and synthetic queries, with high-quality negative mining, target reasoning over hard distractors instead of trivially distinguishable pairs.

Identified limitations and ongoing challenges include the computational cost of listwise or groupwise reranking objectives, which remain GPU-prohibitive for video; dynamic control of computational depth to trade accuracy against latency; and improved handling of queries referencing non-visual events or weakly visual cues.

6. Positioning Within Video Retrieval and Ranking

RankVideo exemplifies the integration of large-scale multimodal transformers and advanced curriculum learning for efficient, high-accuracy reranking in video retrieval. It sets a new empirical standard on event-centric, multilingual benchmarks (MultiVENT 2.0), is agnostic to the choice of first-stage retriever, and advances the state of the art in video-native reasoning reranking (Skow et al., 2 Feb 2026). Its architectural and optimization strategies directly address limitations of both text-only and vision-language reranking alternatives.

For further technical and implementation details, see the original publication (Skow et al., 2 Feb 2026).