Teaching Vision-Language-Action Models What to See and Where to Look

Abstract: Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.

• The paper introduces DriveTeach-VLA, a dual-model framework that uses explicit vision distillation and 2D trajectory-guided prompting to enhance spatial grounding in autonomous driving.
• The methodology leverages a TGP-Prompter and TGP-Planner with block-wise Smooth-L1 alignment and expert trajectory conditioning to optimize planning and improve Attention Mass scores.
• Empirical results on NAVSIM and nuScenes benchmarks demonstrate improved trajectory accuracy and robust performance, validating the framework's practical impact.

Teaching Vision-Language-Action Models What to See and Where to Look: An Expert Analysis

Introduction and Motivation

Vision-Language-Action (VLA) models are increasingly pivotal for end-to-end autonomous driving, promising unified perception, prediction, and planning by leveraging Multi-modal LLMs (MLLMs). However, prevailing VLA frameworks disproportionately rely on visual question answering (VQA) and chain-of-thought (CoT) linguistic data, which fosters strong semantic reasoning but yields spatially ungrounded latent representations. This lack of spatial grounding impedes precise trajectory prediction, as model attention diffusely spans the scene without isolating critical spatial dependencies for navigation and control.

This paper introduces DriveTeach-VLA (2607.01658), proposing an explicit framework to instruct VLA models what to see (injecting driving-relevant perceptual priors) and where to look (spatially guiding future planning), thereby addressing the aforementioned limitations.

Methodology Overview

DriveTeach-VLA utilizes a dual-model architecture: a TGP-Prompter and a TGP-Planner, both built on Qwen2.5-VL-3B. The overarching pipeline is vision-guided: perceptual pretraining (what to see) precedes spatial prompt guidance (where to look), culminating in action planning (how to act).

Figure 1: The DriveTeach-VLA architecture, detailing the interplay between TGP-Prompter for 2D-TGP proposal and TGP-Planner for conditioned action generation.

Driving-aware Vision Distillation (DVD)

Traditional VLM-based encoders lack inductive priors about driving-critical entities (e.g., lanes, dynamic agents). To remedy this, DVD pretraining employs Grounding DINO to detect traffic-critical objects, overlaying bounding boxes onto images as visual prompts. In a self-distillation regime, a teacher ViT, fed the bbox-augmented image, and a student ViT, fed the raw image, undergo latent representation alignment. Block-wise Smooth-L1 alignment across spatial regions ensures the student ViT internalizes traffic saliency independent of explicit textual cues. Empirically, optimal block granularity is $2\times4$.

Figure 2: Schematic of DriveTeach-VLA’s integration of DVD and 2D-TGP for explicit object-based and spatial grounding.

2D Trajectory-Guided Prompting (2D-TGP)

Beyond object-centric induction, spatial conditioning is provided by projecting expert trajectories from BEV to image space via the pinhole camera model. The resultant 2D-TGP encodes a sequence of waypoints that guide the planner towards feasible drivable regions in the visual domain. During training, the TGP-Planner is conditioned on oracle 2D-TGP trajectories (teacher forcing), while at inference, the TGP-Prompter’s predicted 2D-TGP serves as context.

Figure 3: Visualization of predicted 2D-TGP trajectories and critical objects detected by Grounding DINO, exemplifying spatial alignment with intended maneuvers.

TGP-Planner: Spatially Conditioned Reasoning and Planning

Instantiated from the TGP-Prompter’s weights, the TGP-Planner receives the driving scenario (image, ego-state), high-level command, and a textual 2D-TGP prompt. Supervised fine-tuning (SFT) utilizes pseudo-labeled CoT traces and expert demonstrations for stepwise reasoning, culminating in BEV trajectory regression. Reinforcement learning with GRPO further aligns planning with human driving preferences, evaluated using PDMS.

Experimental Results and Analysis

State-of-the-Art Performance

On the NAVSIM (closed-loop) and nuScenes (open-loop) benchmarks, DriveTeach-VLA robustly outperforms prior SoTA baselines across key trajectory prediction and driving safety metrics.

• NAVSIM (PDMS): DriveTeach-VLA attains 90.4, surpassing AutoVLA (89.1). Best-of-N using the DrivorR selector yields a score of 92.7.
• nuScenes (L2 error @ 3s horizon): 0.51m, improving over Impromptu-VLA-3B (0.52m).

DriveTeach-VLA further demonstrates minimal train-test gap: predicted vs. oracle 2D-TGP guidance results in a negligible PDMS difference (0.4 points), supporting the reliability of the promper’s spatial cues.

Qualitative Improvements: Spatial Grounding

DriveTeach-VLA’s attention maps concentrate on bounding boxes of traffic-critical objects, in contrast to diffuse attention from text-centric SFT VLAs.

Figure 4: Attention visualization during decoding: the SFT baseline (left) displays scattered attention, while DriveTeach-VLA (right) focuses on traffic-critical regions.

Quantitatively, the Attention Mass (AM) score of TGP-Planner is significantly higher (mean $3.19 \times 10^{-2}$) than baseline (mean $2.28 \times 10^{-2}$), confirming stronger spatial localization.

Policy Robustness to Guidance Noise

The PDMS remains stable under moderate 2D-TGP prediction errors (see Figure 5), affirming that the trajectory planning module is robust to promper inaccuracies.

Figure 5: PDMS as a function of predicted vs. GT 2D-TGP L2 error, showing policy robustness to moderate spatial guidance deviations.

Ablation: Architectural and Design Choices

Block-wise DVD alignment outperforms patch-wise or larger-grain variants, and moderate loss balancing ($\lambda_\text{TGP}=0.1$) yields optimal joint learning. DVD pretraining and 2D-TGP guidance are both crucial, each incrementally improving mean PDMS by ~1-1.5 points.

Efficiency

Despite introducing runtime overhead via the dual-model design, DriveTeach-VLA’s spatial prompts reduce CoT token requirements. This reduction offsets model invocation costs, yielding competitive runtime (3.18s on L20 GPU) relative to action-token-based AutoVLA.

Theoretical and Practical Implications

• Theoretical: This work substantiates that spatial grounding and explicit perceptual priors are necessary for high-fidelity VLA planning. Object-centric visual distillation and spatially conditioned prompting elevate VLA representations beyond linguistic semantics to actionable spatial intent—a prerequisite for bridging perception-planning gaps in end-to-end autonomy.
• Practical: Integrating DVD and 2D-TGP enables robust, data-efficient transfer from simulation to real-world traffic, due to improved alignment with scene geometry and agent interactions. Robustness to bounding box detection errors also makes the method viable under imperfect perception.

Future Perspectives

Expected research directions include extending spatial prompt modalities (depth-aware, temporal BEV projections), integrating uncertainty estimation in TGP proposals, and further fusing model-based spatial priors with reinforcement learning for adaptive planning under distributional shift. Additionally, scaling to larger MLLM backbones and integrating cross-modal prompt ensembles may yield further gains in both interpretability and robustness.

Conclusion

DriveTeach-VLA defines a new standard for VLA models in autonomous driving by introducing driving-aware vision distillation and spatially explicit trajectory guidance. These innovations realize superior spatial grounding, enabling both state-of-the-art trajectory prediction accuracy and robust, interpretable closed-loop driving performance. Explicitly disentangling perception and planning cues will be central to future vision-language-action frameworks for autonomous systems.