Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation

Abstract: Language-guided UAV agents must execute long-horizon semantic instructions while producing smooth, physically feasible continuous flight commands, yet existing Vision-Language Navigation (VLN) benchmarks typically use discrete or coarse actions and existing UAV Vision-Language-Action (VLA) tasks focus on short, atomic maneuvers. To address this gap in UAV task settings, we introduce \textbf{FLIGHT}, a \textbf{F}ine-grained \textbf{L}ong-horizon \textbf{I}nstruction-\textbf{G}uided benchmark for \textbf{H}ybrid UAV navigation and reasoning \textbf{T}asks, which combines multi-stage instructions with dense 6-DoF trajectory annotations across two dataset splits: Fine-grained VLN and Long-horizon Flow. To endow the UAV agent with the capability of real-time in-flight reasoning over task execution status and mission planning, while simultaneously accommodating high-frequency, real-time precise control, we further propose \textbf{FLIGHT VLA}, an asynchronous architecture that decouples a low-frequency Streaming Pilot Vision-LLM (VLM) for task-state reasoning from a high-frequency diffusion action model for continuous control, supervised by explicit \textbf{Pilot Reasoning} texts that summarize the current flight state and anticipate the next subgoal. In closed-loop evaluation, FLIGHT VLA consistently surpasses representative VLN and VLA baselines on our FLIGHT benchmarks, achieving stronger multi-stage completion, subgoal adherence, and terminal control. Its trained Streaming Pilot Reasoning VLM further improves UAV video reasoning, validating the effectiveness of our design.

• The paper introduces the FLIGHT benchmark, integrating multi-stage semantic instructions with continuous 6-DoF control to tackle long-horizon UAV navigation challenges.
• It presents a dual-rate asynchronous architecture that decouples high-latency semantic processing from fine-grained, real-time control, enhancing navigational stability.
• Empirical results demonstrate significant gains in success rates and instruction adherence over existing VLN and VLA models, validating the pilot reasoning approach.

Fine-Grained Long-Horizon UAV Navigation: The FLIGHT Benchmark and VLA Architecture

Problem Statement and Limitations in Existing Paradigms

The paper "Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation" (2606.06836) systematically analyzes the challenges faced by embodied UAV agents executing long-horizon, language-guided missions. Previous Vision-Language Navigation (VLN) benchmarks exhibit coarse, discrete action spaces and lack the representational fidelity to capture continuous flight dynamics. Conversely, existing Vision-Language-Action (VLA) datasets focus on isolated, short-horizon atomic maneuvers and omit multi-stage compositional instructions, thereby failing to reflect the operational complexity of real-world UAV navigation tasks.

These limitations produce two bottlenecks: (a) an inability to maintain coherent task-state and sub-goal alignment over extended flights due to significant context drift in egocentric visual streams, and (b) an inherent mismatch between the computational latency of high-capacity, semantic VLMs and the real-time demands of closed-loop flight control.

Design and Structure of the FLIGHT Benchmark

FLIGHT introduces a comprehensive benchmark for instruction-guided UAV navigation that fuses multi-stage, semantically rich language instructions with densely annotated, continuous 6-DoF control trajectories. There are two primary task structures:

1. Long-horizon Flow: Extends the UAV-Flow atomic action paradigm by composing multi-step, semantic instructions requiring compositional hybrid actions—e.g., sequentially traversing objects with varying spatial relations.
2. Fine-grained VLN: Densifies referential landmark grounding in instructions and captures tighter action-to-language correspondences, requiring temporally coherent, fine-resolution control that maps closely to real-world UAV requirements.

   Figure 1: Schematic overview of the FLIGHT benchmark, dataset annotation, and dual-rate VLA framework. The left portion contextualizes the hybrid trajectory semantics; the middle details the annotation with Pilot Reasoning supervision; the right illustrates the asynchronous model architecture.

By addressing both multi-stage composition and dense low-level control, FLIGHT rigorously targets hierarchical policy learning in UAV navigation.

Dataset Collection, Annotation, and Pilot Reasoning Paradigm

The FLIGHT dataset is sourced from high-diversity, photorealistic Unrealzoo-based environments and comprises over 10,000 trajectories with continuous motion commands. Unlike prior works, FLIGHT annotates each temporal segment with task-oriented semantic labels, automatically produced by VLMs and enriched by motion priors from UAV telemetry.

The core innovation is the Pilot Reasoning module: for each segment, a Pilot Reasoning text is generated that explicitly encodes both current task status and near-future action plans at decision junctures, mimicking human cognitive flight reasoning. This signal is derived via oracle-assisted annotation involving both current and next-step task labels, thereby supplying dense, chain-of-thought supervision to learning algorithms.

Figure 3: FLIGHT data pipeline: trajectory collection, multimodal VLM-based annotation, and Pilot Reasoning text generation.

This approach enables learned models to align high-level language goals with continuous, temporally grounded flight behaviors while preserving interpretability.

FLIGHT VLA Architecture: Asynchronous Dual-Rate Control

To address temporal-action granularity mismatches, FLIGHT VLA proposes a dual-module asynchronous architecture:

• Streaming Pilot VLM (slow channel): Periodically (at lower frequency), a VLM processes language instructions, FPV video windows, and long-horizon reasoning memory to generate semantic features and pilot reasoning texts. This background module supports sub-task analysis, state estimation, and action intent forecasting.
• Diffusion Action Model (fast channel): At control frequency, visual frames, UAV state, and the latest semantic feature are fused to generate fine-grained action segments via a DiT-based flow matching strategy. The system is robust to VLM latency because the action model interpolates using real-time visual and state cues.

  Figure 5: Asynchronous FLIGHT VLA system: the Streaming Pilot VLM decouples task-state and subgoal reasoning from the high-frequency trajectory generation of the action model.

A critical aspect is the latency-aware joint training protocol: during training, the action model is conditioned on VLM features sampled with stochastic temporal offsets, simulating real-world reasoning delays and promoting policy robustness. This yields stable control even under large, variable-latency language reasoning modules.

Figure 2: Latency-aware asynchronous training results in temporal robustness of the action model to stale or delayed semantic grounding.

Evaluation, Baselines, and Quantitative Results

FLIGHT establishes a rigorous evaluation on two fronts: closed-loop trajectory completion across entire multi-stage missions and adherence to fine-grained instruction decompositions.

Baselines include structurally-adapted SOTA VLN models (CMA+LAG, NaVid) and representative discrete/continuous VLA models (OpenVLA, MemoryVLA), which are augmented to support UAV-specific action spaces and memory requirements.

Empirical results demonstrate:

• On Long-horizon Flow, FLIGHT VLA yields 59.0% SR, 78.97% SSR, and 45.78% NDTW, substantially outperforming OpenVLA (SR: 10.50%) and MemoryVLA (SR: 18.50%).
• On Fine-grained VLN, while absolute SR remains low due to the difficulty, FLIGHT VLA nearly doubles the Instruction Adherence SR over baselines, and improves navigation error and OSR.
• The trained Streaming Pilot Reasoning VLM achieves state/action reasoning accuracies of 64.2%/68.1%—vastly superior to existing LLMs or video foundation models on complex UAV FPV data.

  Figure 8: Policy rollout visualizations: the UAV accurately follows multi-stage instructions in both compositional flow and fine-grained VLN conditions, demonstrating subgoal adherence and effective task-state reasoning.

These findings establish the necessity and efficacy of (a) asynchronous semantic/motor control and (b) explicit temporal Pilot Reasoning supervision for high-fidelity language-to-trajectory policies.

Implications, Limitations, and Future Directions

Practically, the integration of a streaming reasoning backbone with a diffusion-based action model directly improves the deployability of language-driven UAV agents in real-world, latency-constrained settings. The Pilot Reasoning signal constitutes a strong, interpretable supervisory signal, potentially facilitating robust model-based or reinforcement learning extensions.

Theoretically, the formalization of explicit intermediate reasoning and hierarchical decoupling offers a template for embodied AI across modalities requiring tight coupling of long-horizon semantic memory and low-latency action, including ground vehicles and legged robotics.

However, the scope remains simulation-centric, and extension to multi-UAV/multi-agent realms, dynamic physical constraints, or zero-shot generalization to unseen semantic environments is nontrivial. Future developments may include RL fine-tuning, autonomous error recovery, unifying visual memory architectures (Torne et al., 4 Mar 2026), or open-world reasoning extensions (Intelligence et al., 22 Apr 2025, Zhou et al., 28 May 2025).

Conclusion

FLIGHT and the FLIGHT VLA framework collectively advance the state of language-conditioned UAV navigation by fusing multi-scale grounded instruction-following, chain-of-thought pilot reasoning, and robust asynchronous architecture for long-horizon, fine-grained control. The results reinforce the necessity of explicit intermediate reasoning and dual-rate architectures in aligning high-level semantic understanding with physically plausible, continuous robot policy learning. The benchmark and associated methods set a new standard for evaluation and design of UAV navigation agents under realistic, instruction-guided regimes.