DriveReward: A Comprehensive Dataset and Generative Vision-Language Reward Model for Autonomous Driving

Abstract: Reward models play a pivotal role in reinforcement learning (RL) and multi-modal trajectory selection for autonomous driving. However, acquiring such rewards typically relies on hand-crafted rule-based objectives or perception ground truth, which hinders generalization for data-scaling. While Vision-LLMs (VLMs) have demonstrated feasibility as reward models in other domains, their effectiveness in driving tasks remains underexplored. In this work, we bridge this gap by (1) introducing DriveReward, a reasoning trajectory evaluation dataset rigorously labeled via temporally-grounded visual guidance, and augmented with counterfactual driving behaviors., (2) alongside a specialized Vision-Language Reward Model. To address the scarcity of failure cases in conventional datasets, we propose a counterfactual data annotation scheme to construct cases encompassing diverse driving styles and erroneous behaviors. Evaluations on our proposed benchmark reveal that even leading open-source and proprietary VLMs fail to excel across all tasks, highlighting significant room for improvement in existing models. Building on these findings, we subsequently tailor a specialized 1B reward model that outperforms larger VLMs on task-specific reward alignment. Finally, we validate our reward model's effectiveness by integrating it into RL finetuning and multi-modal trajectory scoring across multiple baselines, achieving performance comparable to rule-based reward calculations in both open-loop and closed-loop evaluation.

• The paper introduces a comprehensive dataset and generative vision-language reward model for autonomous driving, reducing reliance on manual perception labels.
• It employs counterfactual trajectory evaluation and specialized metrics across seven dimensions, including non-collision and legality.
• Empirical results demonstrate enhanced reinforcement learning performance and state-of-the-art multi-dimensional reward alignment for trajectory selection.

DriveReward: A Generative Vision-Language Reward Model and Dataset for Autonomous Driving

Motivation and Background

Autonomous driving research has progressed to planning paradigms that leverage multi-modal trajectory generation, aiming to address inherent uncertainty and diversity in real-world driving scenarios. Existing evaluation methodologies for candidate trajectories, essential for both RL and test-time inference, are typically either model-free (rule-based, heavily dependent on perception outputs and hand-crafted metrics) or model-based (jointly trained selector heads or world models optimized for proxy metrics such as MSE or PDMS). These methods lack high-level semantic reasoning and generalization ability, particularly in the presence of long-tail events and complex traffic rule violations. Vision-LLMs (VLMs), which have recently demonstrated potential in constructing reward functions in other domains, remain underutilized in automotive contexts.

The DriveReward project introduces two primary contributions addressing these gaps: (i) the DriveReward dataset and benchmark, focused on temporally-grounded, counterfactual trajectory evaluation; (ii) the DriveReward-1B generative VLM reward model, engineered for robust, multi-dimensional trajectory assessment without explicit perception annotation dependencies.

Figure 1: Overview of the DriveReward system highlighting the holistic reward modeling process, dataset curation pipeline, and architectural design.

Dataset Construction and Benchmark

The DriveReward dataset is architected to overcome the sparsity and imbalance seen in conventional driving datasets, where nominal and safe driving behaviors dominate, hindering discriminative reward modeling. Negative-example data augmentation strategically synthesizes long-tail infraction scenarios—including failures, unsafe maneuvers, and rule violations. The annotation pipeline systematically aggregates trajectory candidates from ground truth, offline kinematic trajectory banks, and planner outputs, with subsequent labeling across seven dimensions: Non-Collision (NC), Drivable Area Compliance (DAC), Ego Progress (EP), Time-to-Collision (TTC), Comfort (C), Command Following (CF), and Legality (LG).

Specialized evaluation metrics for CF and LG are introduced to address weak command adherence and traffic regulation infractions, respectively. CF is determined via waypoint-based deviation from ground-truth, and LG is assessed using a large VLM (Qwen3.5-35B-A3B) for red-light and solid-lane violations with tiered scoring.

Trajectory candidates are visually and temporally grounded with overlayed prompts on images, facilitating nuanced, fine-grained VLM annotation.

Figure 2: Comparative visualization of model-free, model-based, and the DriveReward paradigm for trajectory evaluation.

Figure 3: Automated pipeline for labeling trajectories leveraging multi-source candidates and multi-dimensional evaluation.

The benchmark, DriveReward-Bench, established on the NAVSIM test set, evaluates reward model performance using five critical dimensions—NC, DAC, CF, LG, and EP—excluding TTC and Comfort due to their redundancy and limited discriminative value in sampled data.

Figure 4: Distribution of sub-metrics and trajectory scores in the DriveReward dataset, illustrating enriched failure and infraction cases.

Figure 5: DriveReward-Bench benchmark distribution, providing coverage for diverse evaluation dimensions.

DriveReward Model Design and Training

DriveReward-1B, built atop InternVL3-1B, incorporates geometry grounding to enforce spatial comprehension, and outputs reward reasoning and geometry latent tokens for direct supervision. The multi-modal input consists of the current front-view images, navigational instructions, ego state vectors, and candidate trajectory data. The model generates Chain-of-Thought (CoT) reasoning and structured numerical scores in a generative, autoregressive manner.

Geometry grounding is realized by matching output features to VGGT latents using an MLP-based adapter, supervised via $L_2$ loss, enabling explicit spatial constraint learning without dense 3D annotation.

Figure 6: The DriveReward model architecture, depicting multimodal input processing, geometry grounding, reward/CoT output and the two-stage training recipe.

The training protocol is two-stage: (Stage 1) domain-specific pre-training on aggregated QA datasets for scene understanding, and (Stage 2) supervised fine-tuning (SFT) on the DriveReward dataset for multi-dimensional trajectory assessment. The annotation process strictly utilizes original rather than visually prompted images.

Figure 7: Reasoning data generation pipeline, detailing VLM-based annotation for trajectory assessment and rule compliance.

Empirical Results and Quantitative Evaluation

DriveReward-1B demonstrates substantial performance improvements across all discrete evaluation dimensions, outperforming larger open-source models (e.g., Qwen3, Qwen3.5, InternVL3) and strong SFT baselines. For Safety, DAC, CF, and Legality, the model achieves up to 99.9% accuracy, with EP-MAE at 0.23. Notably, targeted reward supervision enables the lightweight 1B model to surpass much larger general-purpose VLMs.

RL experiments reveal that DriveReward-1B-predicted reward signals (PDMS) produce consistent gains in both open-loop and closed-loop policy performance. Augmenting traditional rule-based PDMS with model-predicted CF and LG scores further enhances safety and compliance, albeit with minor efficiency tradeoffs. Closed-loop evaluations on Bench2Drive show strong zero-shot generalization, with reward model-based RL outperforming baseline SFT, and composite reward models pushing performance to the highest reported Driving Score (DS).

Test-time trajectory selection on AdaThinkDrive verifies that reward-based selection can partially capture Best-of-N potential, though a notable gap remains, suggesting future avenues for improving multi-modal discrimination.

Ablation Analyses

Ablation studies isolate contributions of the two-stage training, CoT reasoning, and geometry adapter modules. Domain-specific pre-training and explicit spatial grounding are confirmed to provide consistent gains, while removal of CoT significantly degrades model capacity for semantic trajectory evaluation.

Limitations and Future Directions

Current dataset and modeling are limited to single front-view imagery, excluding richer modalities such as LiDAR or multi-camera fusion. Multi-sensor integration and increased temporal context remain essential for robust generalist reward modeling. The domain gap between public datasets and real-world deployment—owing to lack of fine-grained routing, strategic instructions, and multi-modal interaction—poses further challenges that need to be addressed.

Practical and Theoretical Implications

DriveReward presents a scalable solution to trajectory evaluation, eliminating dependency on explicit perception labels and enhancing adaptability to unannotated or instruction-driven domains. The generative approach to reward reasoning, combined with multi-dimensional evaluation, opens avenues for broader VLM-based reward modeling in other sequential decision-making tasks and complex interactive environments.

Theoretically, this work bridges the gap between interpretability and efficiency in reward modeling, demonstrating that robust reasoning supervision and explicit geometric grounding amplify the semantic and spatial acuity of vision-language reward evaluators. It sets the stage for future RL alignment techniques and trajectory selection frameworks using lightweight, specialized vision-language architectures.

Figure 8: Visualization of DriveReward samples with video sequence input, ego-state condition, and multi-dimensional annotation.

Conclusion

DriveReward establishes a unified trajectory evaluation and reward modeling pipeline for autonomous driving, leveraging generative vision-LLMs trained with counterfactual, temporally-grounded datasets. The DriveReward-1B model achieves state-of-the-art multi-dimensional reward alignment, delivers robust RL improvements, and exhibits strong generalization across test-time trajectory selection and simulation benchmarks. Future research should focus on multi-modal sensor fusion and bridging offline/online domain gaps to enable deployment-ready reward modeling paradigms in real-world autonomous driving (2606.08525).