Streaming Video Question Answering

• Streaming Video QA is a task that processes continuous video data in real time, generating answers by incrementally analyzing incoming frames along with relevant historical context.
• Systems employ adaptive memory management, dynamic segmentation, and attention-based token selection to efficiently handle long video streams with minimal computational overhead.
• Evaluation focuses on temporal localization, causal reasoning, and latency management, driving advances for practical applications like surveillance, interactive assistants, and live broadcast analysis.

Streaming video question answering (Streaming Video QA) refers to the task of automatically generating answers to natural language questions about ongoing, potentially unbounded video streams in real time. Unlike traditional video QA, where answers are inferred from a fully observed, temporally static video segment, streaming video QA systems must incrementally process incoming frames, maintain relevant historical context, adapt to asynchronous user queries, and deliver responses with minimal computational latency and memory overhead. This paradigm is critical for applications in surveillance analytics, embodied agents, live broadcast understanding, and interactive assistants in egocentric or dynamic environments.

1. Formal Problem Definition and Distinguishing Characteristics

At its core, streaming video QA poses a temporally grounded reasoning challenge: Let a video stream be represented as a sequence of frames $\{f_t\}_{t=1}^\infty$ with $t$ indexing arrival time. A user can pose a question $q$ at any time, and the system must answer $a$ by leveraging both the current and relevant past visual and (optionally) dialogue history, subject to real-time constraints. Models must:

• Process incrementally and online, with future frames unavailable at query time.
• Retain and select from potentially long historical streams, under bounded memory/compute.
• Manage asynchronous, multi-turn dialogue and referential chains.
• Localize events temporally and reason about causality, progression, and spatiotemporal dependencies.

Streaming video QA encompasses sub-problems of event detection, temporal localization, multimodal alignment, selective memory management, efficient context retrieval, and real-time response under uncertainty (Yang et al., 15 Feb 2025, Kong et al., 2023, Hu et al., 29 Oct 2025, Zhao et al., 12 Jun 2025).

2. Datasets and Benchmarking Frameworks

Contemporary progress has been catalyzed by purpose-built benchmarks that simulate streaming constraints, multi-turn dialogues, dense temporal annotation, and embodied scenarios:

Benchmark	QA Pairs	Temporal Structure	Key Features	Reference
SVBench	49,979	Multi-turn, overlapping	Temporal QA chains, LLM–human curation	(Yang et al., 15 Feb 2025)
ATBS	243,680	Target-in-long background	Online localization, O$^2$VQA setting	(Kong et al., 2023)
StreamingCoT	68,940	Per-second & segments	Multimodal CoT chains, state tracking	(Hu et al., 29 Oct 2025)
CogStream	59,032	Hierarchical, contextual	Context-guided retrieval, event-aware	(Zhao et al., 12 Jun 2025)
VStream-QA	3,500	1h egocentric/movie	Latency/Vram tracking, multi-QA types	(Zhang et al., 2024)
MMDuetIT	109,000	Arbitrary streaming turns	Duet interaction, time-sensitive QA	(Wang et al., 2024)
StreamEQA	20,731	Embodied, streaming mode	Perception/interaction/planning axes	(Wang et al., 4 Dec 2025)

Key innovations include hierarchical annotation (per-second dense, event-based), dynamic segment construction via similarity fusion, chain-of-thought (CoT) annotations for logical reasoning, and fine-grained temporal linkage across QA pairs. Evaluation protocols emphasize not only answer correctness but also context retention, temporal/causal reasoning, latency, and memory footprint under streaming conditions.

3. Modeling Architectures and Memory Mechanisms

Streaming video QA systems have evolved sophisticated mechanisms to encode, compress, and retrieve long-range visual and textual context:

• Segmentation, Compression, and Memory Propagation: Dynamic segmentation of streams into variable-length, semantically-coherent intervals is standard for memory efficiency. Both StreamKV and StreamingCoT use segment-wise embeddings with summary tokens and semantic similarity thresholds for content grouping; Flash-VStream employs a STAR memory decomposed into spatial, temporal, abstract, and retrieved slots, maintaining constant memory size regardless of stream length (Chen et al., 10 Nov 2025, Yang et al., 15 Feb 2025, Zhang et al., 2024, Hu et al., 29 Oct 2025).
• Memory-Adaptive Selection and Storage: Models such as VideoStreaming and StreamKV use memory-propagated streaming encoding, maintaining and updating a compact memory at each timestep, and, upon querying, retrieve only the most question-relevant memory blocks as determined by similarity scoring (Qian et al., 2024, Chen et al., 10 Nov 2025). ReKV extends this to large-scale systems by multi-tier storing of transformer KV-caches (GPU, RAM, disk) and selective, question-sensitive retrieval (Di et al., 1 Mar 2025).
• Attention-Based Token Selection: Recurrent attention-based selection (e.g., (Dorovatas et al., 20 Oct 2025)) prunes up to 95% of visual tokens per clip, guided by cross-attention weights to generated captions, and recurrently propagates only attended, salient tokens.
• Dialogue and Event Selection: CogReasoner employs K-means clustering for temporal-semantic event pooling and a fine-tuned LLM-based historical dialogue retriever, dynamically selecting only contextually helpful QA pairs (Zhao et al., 12 Jun 2025).
• Real-Time Triggering and Duet Format: Time-sensitive models (e.g., MMDuet (Wang et al., 2024)) employ per-frame "informative" and "relevance" heads to autonomously trigger responses within a continuous video–text duet, enabling frame-wise intervention and multi-answer outputs as events unfold.

4. Evaluation Methodologies and Metrics

Streaming video QA evaluation leverages a combination of classical and streaming-specific criteria:

• Traditional Metrics: Top-1 accuracy, F1, MRR, METEOR, ROUGE-L, and CIDEr, often with semantic alignment refinements (e.g., GPT4-Score) (Yang et al., 15 Feb 2025).
• LLM-Driven Judging: Outputs are scored along axes such as Semantic Accuracy (SA), Contextual Coherence (CC), Logical Consistency (LC), Temporal Understanding (TU), Informational Completeness (IC), with overall scores reflecting a composite of these properties (Yang et al., 15 Feb 2025, Zhang et al., 2024).
• Latency and Efficiency: Metrics include end-to-end QA latency, VRAM consumption, memory usage relative to stream length, and throughput (frames per second); ablations quantify the tradeoffs between retained context and computational resource demands (Zhang et al., 2024, Chen et al., 10 Nov 2025, Di et al., 1 Mar 2025).
• Temporal and Embodied Reasoning: Advanced benchmarks (e.g., StreamEQA (Wang et al., 4 Dec 2025)) stratify tasks by reasoning mode (backward, real-time, forward), embodied task type (perception, interaction, planning), and measure performance gaps among axes—forward temporal reasoning and high-level planning remain major bottlenecks.
• Context Relevance and Robustness: Retrieval-based models are assessed on sensitivity to irrelevant or noisy context, resilience to distractor QA/history, and explicit measurement of context selection accuracy (e.g., RQS F1 in CogReasoner) (Zhao et al., 12 Jun 2025).

5. State of the Art: Model Comparison and Technical Trends

Empirical results from recent benchmarks reveal a consistent pattern:

• Closed-source LVLMs (e.g., GPT-4o) maintain a robust but narrowing lead over open-source alternatives (e.g., StreamingChat, Flash-VStream, CogReasoner), especially as streaming length and dialogue complexity scale up (Yang et al., 15 Feb 2025, Zhang et al., 2024).
• Open-source systems adopting segment-based compression, retrieval-guided memory, attention-based token selection, and streaming-aware training (e.g., StreamingChat, StreamKV, CogReasoner) close the performance gap under real-time, resource-constrained regimes (Chen et al., 10 Nov 2025, Zhao et al., 12 Jun 2025, Yang et al., 15 Feb 2025).
• Table: Key Model Scores (overall, OS out of 100, or Acc/Score where noted):

Model	Streaming Score	Notable Features	Ref
GPT-4o	58.17 (SVBench OS)	Closed, long-context	(Yang et al., 15 Feb 2025)
StreamingChat	53.90 (SVBench OS)	Open, InternViT+InternLM2	(Yang et al., 15 Feb 2025)
StreamKV	58.9% (StreamingBench)	KV-cache segment compression	(Chen et al., 10 Nov 2025)
Flash-VStream	53.1% (VStream-QA)	STAR memory, real-time	(Zhang et al., 2024)
CogReasoner	72.26 (CogStream avg)	Compress & retrieve dialogue/video	(Zhao et al., 12 Jun 2025)

Gaps between offline, full-access models and their streaming/online variants remain pronounced, with substantial degradation in performance for forward prediction and high-level planning subtasks. Robust context selection and long-term memory management are repeatedly highlighted as urgent open challenges.

6. Challenges, Limitations, and Prospects

Persistent open problems include:

• Long-Horizon Memory and Retrieval: Transformers degrade after ~100–200 frames without explicit memory mechanisms. Methods such as hierarchical/sliding-window attention, explicit long-term memory slots, and adaptive retrieval budgets have been partially successful; dynamic quantization and adaptive compression remain active research directions (Yang et al., 15 Feb 2025, Di et al., 1 Mar 2025, Chen et al., 10 Nov 2025).
• Temporal Drift and Nonlinear Context: Models often exhibit recency bias, with sharp declines in contextual coherence and temporal understanding over long contexts or under out-of-order (jump) evaluations (Yang et al., 15 Feb 2025).
• Event / Dialogue Disambiguation: The identification of question-relevant events/dialogue remains a bottleneck; even state-of-the-art retrieval modules have imperfect precision, and model robustness to injected dialogue noise is variable (Zhao et al., 12 Jun 2025).
• Forward Reasoning and Planning: In embodied QA, forward prediction lags perception/recall by 10–20 percentage points, reflecting fundamental limitations in anticipating future actions from partial context (Wang et al., 4 Dec 2025).

Recommendations (as compiled from SVBench, CogStream, StreamKV):

• Introduce hierarchical or retrieval-augmented memory management.
• Leverage adaptive, semantic-aware segmentation rather than uniform sampling to optimize context granularity.
• Explicitly signal and track dialogue and event linkages via “chain-aware” prompts and auxiliary objectives.
• Employ joint multimodal reasoning with both compressed visual and filtered dialogue context.
• Align training curricula with growing temporal horizons and increasing task complexity.

7. Impact, Applications, and Future Directions

Streaming video QA research is foundational for next-generation systems operating in live, open-world settings—autonomous robotics, surveillance, augmented reality, assistive devices, interactive media analysis, and real-time situational comprehension. The ongoing development of benchmarks (e.g., StreamingCoT, StreamEQA) and increasingly sophisticated retrieval, compression, and memory-propagation architectures provides quantitative diagnostic tools and a clear roadmap for advancing robust, context-aware, and efficient real-time video understanding.

Continued progress is expected along the axes of:

• Explicit memory-controller architectures for very long streams.
• End-to-end differentiable context selection integrating user intent, event salience, and temporal history.
• Multimodal chain-of-thought reasoning paired with interpretable, auditable outputs.
• Scalable, plug-in modules for downstream embodied intelligence and multi-agent coordination in persistent, dynamic environments.

The field is converging on hybrid paradigms that unify event-centric video encoding, adaptive retrieval, context-guided selection, and dialogue-interleaved reasoning. These advances collectively delineate the emerging paradigm of streaming video question answering. (Yang et al., 15 Feb 2025, Deng et al., 2023, Chen et al., 10 Nov 2025, Zhao et al., 12 Jun 2025, Qian et al., 2024, Zhang et al., 2024, Wang et al., 2024, Dorovatas et al., 20 Oct 2025, Hu et al., 29 Oct 2025, Wang et al., 4 Dec 2025)