Ego-R1: Ultra-Long Egocentric Video Reasoning

• Ego-R1 is a framework for ultra-long egocentric video reasoning that decomposes complex queries into modular steps using a Chain-of-Tool-Thought process.
• It employs a hierarchical retrieval system alongside specialized visual tools like video-LLM and VLM to locate and analyze evidence over extended time periods.
• The framework integrates supervised fine-tuning with reinforcement learning to optimize tool planning, enhancing interpretability and QA performance.

Searching arXiv for the specified papers and closely related work on Ego-R1. Ego-R1 is a framework for reasoning over ultra-long egocentric videos, defined in the 2025 paper "Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning" as a system that leverages a structured Chain-of-Tool-Thought (CoTT) process orchestrated by an Ego-R1 Agent trained via reinforcement learning (Tian et al., 16 Jun 2025). It targets first-person videos that span days to a week, where answering a query at a given time often depends on sparse, temporally distant evidence and enduring personal routines. In this framework, complex questions are decomposed into modular steps, and an LLM-based orchestrator dynamically invokes specialized tools, one per step, to retrieve, localize, and analyze evidence before producing an answer. The framework is designed to extend time coverage from few hours to a week while preserving interpretability (Tian et al., 16 Jun 2025).

1. Problem setting and conceptual motivation

Ego-R1 addresses ultra-long egocentric videos in which relevant evidence may be distributed over hours, days, or weeks rather than within a single short clip (Tian et al., 16 Jun 2025). The formal setting constrains the system to use only content prior to the query timestamp. This temporal causality requirement is central: the question is posed at a specific query time, and any reasoning chain must be grounded exclusively in pre-query observations.

The framework is motivated by several limitations of prior approaches. Feeding entire weeks of video into MLLMs is infeasible; token compression and uniform sampling can discard rare but decisive events; and prior video agents often rely on fixed tool orders or pre-defined pipelines that do not adapt on the fly (Tian et al., 16 Jun 2025). Ego-R1 therefore adopts dynamic tool selection rather than a static perception pipeline.

CoTT is the organizing principle. The framework is explicitly inspired by human problem-solving strategies: break a task into sub-questions, determine what evidence is needed next, call the appropriate tool, and integrate observations over time. This design is intended to reduce unnecessary perception, preserve temporal awareness, and improve interpretability. A plausible implication is that Ego-R1 treats long-horizon egocentric reasoning primarily as an evidence-navigation problem rather than only as a long-context sequence modeling problem.

2. Chain-of-Tool-Thought formalism

CoTT defines a multi-step trajectory

$$C = (S^0, S^1, \dots, S^n), \qquad S^i = (T^{\mathrm{th}_i}, T^{\mathrm{to}_i}, o_i),$$

where at each step $i$ the agent produces a thought $T^{\mathrm{th}_i}$ and a tool call $T^{\mathrm{to}_i}$ conditioned on the question $q$ and prior observations, and the tool execution returns an observation $o_i$ that guides the next step (Tian et al., 16 Jun 2025).

The action space is defined over specialized tools,

$A = \{F_j\},$

and includes h-rag, video-LLM, and vlm, plus an auxiliary terminate tool used only during data generation (Tian et al., 16 Jun 2025). The observation space is typed and structured:

$o_i = (o_i^{rag}, o_i^{vid}, o_i^{vlm}) \in O.$

Each observation is fed back to the agent, which then decides the next thought and next tool.

A defining property of CoTT is the constraint of one tool per step. This yields an explicit reasoning trace in which each stage has a distinct epistemic role: retrieval, localized video analysis, or frame-level inspection. The runtime interface enforces function-call syntax. The three main tool signatures are:

• rag: arguments $$\{ \text{level} \in \{\text{week}|\text{day}|\text{hour}\}, \text{keywords}: \text{List[str]}, \text{start\_time}, \text{query\_time} \}$$
• video_llm: arguments $\{ \text{question}: \text{str}, \text{range} \}$ with $i$0 sec $i$1 length $i$2 min
• vlm: arguments $i$3 (Tian et al., 16 Jun 2025)

Every step starts with > …</think>, then either issues a <tool>…</tool> call and consumes the returned <information>…</information>, or concludes with <answer>…</answer> once sufficient evidence has been assembled. This explicit trajectory structure is one of the framework’s main interpretability features.

3. Tool stack, memory hierarchy, and data resources

Ego-R1 couples a hierarchical retrieval system with two visual analysis tools (Tian et al., 16 Jun 2025). The retrieval component, H-RAG, builds a memory bank across granularities. Raw video is segmented into 30-second clips; each clip is captioned and aligned with ASR to form clip logs; bottom-up aggregation then produces multi-level summaries such as 10-minute, hour, and day summaries while retaining timestamps and metadata. At inference time, top-down retrieval selects relevant time ranges and summaries using the specified level, keywords, and time constraints, then drills down when needed.

The visual tools operate at different temporal resolutions. Video-LLM performs short-horizon visual reasoning over sampled clips at 1 FPS and supports windows from seconds to at most 10 minutes. VLM performs single-frame analysis for high-resolution details such as text on packaging or fine-grained attributes (Tian et al., 16 Jun 2025). The orchestrator itself, named Ego-ReaLM, remains in the language space and queries visual evidence through tools rather than directly ingesting week-long visual streams.

The modularity of the tool stack is explicit. Visual backbones can be swapped, with the paper noting, for example, LLaVA-Video and Gemini-1.5-Pro as interchangeable perception components (Tian et al., 16 Jun 2025). This suggests that the framework is architected as an agentic control layer over heterogeneous perceptual subsystems rather than as a monolithic multimodal model.

Training relies on Ego-R1 Data, which consists of two datasets (Tian et al., 16 Jun 2025):

• Ego-QA-4.4K: 4.4K human and synthetic QA pairs from approximately 500 hours of egocentric videos across six participants.
• Ego-CoTT-25K: 25K synthetic CoTT traces built on 2.9K selected high-quality questions, with each trace averaging 7.42 tool calls.

For supervised data generation, the long-context model outputs <think>… followed by <tool>…</tool> or <answer>…</answer>, arguments are validated and executed, and observations are fed back for the next step. The resulting traces are tool-grounded rather than purely textual.

4. Two-stage training paradigm

Ego-R1 uses a two-stage training pipeline: supervised fine-tuning followed by reinforcement learning (Tian et al., 16 Jun 2025). In Stage 1, the base model is Qwen-2.5-3B-Instruct, which becomes Ego-R1-SFT after finetuning. The SFT stage teaches the agent to produce correctly formatted tool calls, follow the CoTT schema, and maintain coherent step-wise reasoning. The training format explicitly simulates multi-turn tool use with <think>, <tool>, <information>, and <answer> tags.

Stage 2 applies RL with GRPO. The objective is described as maximizing expected final task reward while regularizing policy drift and stabilizing gradients across steps through group-wise standardization (Tian et al., 16 Jun 2025). The paper gives the GRPO objective as $i$4, with rollout groups $i$5, clipped policy-ratio terms, standardized advantages $i$6, and a KL regularizer $i$7. The rollout procedure iteratively generates thoughts and tool calls, executes tools, injects observations, and terminates on a valid final answer or a step limit $i$8.

The reward design is rule-based at the final task level. Answer correctness supplies the task reward, while stability is encouraged through KL regularization and group-relative advantages (Tian et al., 16 Jun 2025). The training rationale is summarized in the paper’s own ablation interpretation: “SFT memorizes, RL generalizes.” In that formulation, SFT provides structured tool use and format compliance, whereas RL improves multi-turn decision making and adaptive tool selection.

A notable feature of Ego-R1 is that RL is not used to replace the tool interface; it is used to optimize planning over the interface. This distinguishes the framework from approaches that treat reasoning as direct end-to-end sequence generation over fixed video tokens.

5. Evaluation, benchmarks, and empirical behavior

The principal benchmark for the framework is EgoR1QA (Ego-R1 Bench), a week-long, human-verified QA benchmark containing 300 QA pairs evenly distributed across six first-person perspectives (Tian et al., 16 Jun 2025). For each perspective, it includes a balanced mix of human-labeled and human-verified synthetic questions, all constrained to use only pre-query content. The average per-question temporal span is approximately 44.3 hours, making the benchmark explicitly week-scale.

Ego-R1 is also evaluated on VideoMME, EgoSchema, and EgoLifeQA under causal constraints or cleaned evaluation settings (Tian et al., 16 Jun 2025). Reported accuracies are:

• EgoR1QA: 46.0%, surpassing Gemini-1.5-Pro at 38.3% and LLaVA-Video at 29.0%
• VideoMME: 64.9%, second to Gemini-1.5-Pro at 67.4%
• EgoSchema: 68.2%, second to Gemini-1.5-Pro at 72.2%
• EgoLifeQA: 36.0%, comparable to LLaVA-Video at 36.4% and approaching Gemini-1.5-Pro at 36.9% (Tian et al., 16 Jun 2025)

The ablation study is central to understanding the framework. On EgoR1QA, the zero-shot base model achieves 1.4% task accuracy and 4.3% format accuracy; RL-only reduces task accuracy to 0% while improving format accuracy by 9%; SFT-only reaches 34.3% task accuracy and 100% format accuracy; and SFT+RL reaches 46.0% task accuracy and 100% format accuracy (Tian et al., 16 Jun 2025). These results are used to argue that CoTT supervision is necessary before RL can refine tool planning effectively.

Tool-level ablations further show that replacing LLaVA-Video with Gemini-1.5-Pro improves EgoR1QA from 43.7% to 46.0%, demonstrating modularity, while a retrieval-only configuration reduces performance to 39.7%, indicating that hierarchical retrieval and visual grounding are jointly necessary (Tian et al., 16 Jun 2025). Qualitative analysis identifies interpretable step-by-step traces as a strength and failure to adjust temporal ranges after a promising retrieval as a recurrent failure mode.

6. Relation to adjacent “Ego-R1-style” usages

The term Ego-R1 is used explicitly for the CoTT-based ultra-long reasoning framework introduced in 2025 (Tian et al., 16 Jun 2025). However, adjacent literature includes systems that are described as “Ego-R1-style” in a broader or different sense.

The 2024 paper GEM does not mention “Ego-R1” and does not present GEM as Ego-R1 or as a named precursor (Hassan et al., 2024). It is instead a Generalizable Ego-vision Multimodal world model that predicts future frames from a single reference frame while conditioning on ego-trajectories, sparse DINOv2 tokens, and human pose skeletons. The paper’s summary states that GEM can reasonably be interpreted as an Ego-R1-style framework insofar as it is an ego-centric, controllable, multimodal world model with fine-grained control over future egocentric observations, paired RGB-depth outputs, and an autoregressive sampling scheme for long-horizon generation (Hassan et al., 2024). This is nevertheless a different technical objective: controllable future synthesis rather than question answering over ultra-long memory.

The 2025 paper EgoVLM uses an “R1-style” reinforcement learning procedure, specifically GRPO, for egocentric video understanding without supervised chain-of-thought pretraining (Vinod et al., 3 Jun 2025). It is described as bringing an R1-Zero-like RL training scheme to egocentric video QA: sampled frames, explicit <think> and <answer> traces, and rule-based rewards on format and correctness. Unlike Ego-R1, EgoVLM does not center on hierarchical retrieval over week-long videos or dynamic tool calling; it operates as a policy-optimized VLM over sampled egocentric frames (Vinod et al., 3 Jun 2025).

A still earlier and unrelated use appears in the 2010 visualization framework for large repositories, where “Ego-R1 style” refers to radius-1 ego graphs in a time-aware exploratory interface (Reitz, 2010). There, “R1” denotes one-hop ego networks rather than reinforcement-learning-based reasoning or egocentric video analysis. The coexistence of these usages makes terminology potentially ambiguous. In current egocentric video research, however, Ego-R1 most specifically denotes the 2025 Chain-of-Tool-Thought framework for ultra-long egocentric video reasoning (Tian et al., 16 Jun 2025).