Hidden Biases of End-to-End Driving Datasets

Abstract: End-to-end driving systems have made rapid progress, but have so far not been applied to the challenging new CARLA Leaderboard 2.0. Further, while there is a large body of literature on end-to-end architectures and training strategies, the impact of the training dataset is often overlooked. In this work, we make a first attempt at end-to-end driving for Leaderboard 2.0. Instead of investigating architectures, we systematically analyze the training dataset, leading to new insights: (1) Expert style significantly affects downstream policy performance. (2) In complex data sets, the frames should not be weighted on the basis of simplistic criteria such as class frequencies. (3) Instead, estimating whether a frame changes the target labels compared to previous frames can reduce the size of the dataset without removing important information. By incorporating these findings, our model ranks first and second respectively on the map and sensors tracks of the 2024 CARLA Challenge, and sets a new state-of-the-art on the Bench2Drive test routes. Finally, we uncover a design flaw in the current evaluation metrics and propose a modification for future challenges. Our dataset, code, and pre-trained models are publicly available at https://github.com/autonomousvision/carla_garage.

• The paper reveals that the expert driver's 'style' significantly impacts imitation learning model performance in end-to-end driving datasets.
• Frequency-based data weighting, commonly used in dataset design, can detrimentally skew predictions in scenarios with diverse behaviors.
• A novel method filtering frames by target label changes allows for halving the dataset size while maintaining predictive performance.

An Analytical Overview of "Hidden Biases of End-to-End Driving Datasets"

The paper "Hidden Biases of End-to-End Driving Datasets" addresses several crucial considerations and challenges in the field of imitation learning (IL) for end-to-end autonomous driving systems, particularly focusing on the oppressive CARLA Leaderboard 2.0. The authors seek to explore an often-overlooked aspect of autonomous driving datasets: the inherent biases and nuances that can significantly affect the performance of IL policies.

Key Insights

1. Importance of Expert Style: An intriguing finding of the study is that the 'style' of the expert driver used for generating training data critically influences the downstream performance of imitation learning models. Factors like how the expert reacts to obstacles and its braking behavior materially impact the model's ability to generalize to new situations.
2. Data Weighting Considerations: The paper challenges conventional practices in IL dataset design, such as the application of frequency-based class weights. It posits that relying on these methods can detrimentally skew predictions in scenarios like target speed forecasting where over-represented classes might contain significant behavioral diversity.
3. Efficient Dataset Reductions: The authors propose a novel tactic of filtering datasets by assessing frame importance based on changes in target labels. This approach enabled them to halve their dataset size without sacrificing predictive performance, enhancing computational efficiency in model training.
4. Metric Evaluation and Proposal for Improvement: A methodological critique within the paper highlights a flaw in the current evaluation metrics that could incentivize premature termination of test scenarios, potentially skewing results. The authors suggest a shift towards normalized metrics that encourage full completion of test routes, supporting more reliable performance evaluation.

Practical and Theoretical Implications

Practically, these findings reshape the approach towards designing and evaluating training datasets for autonomous driving, promoting robust and generalizable IL policies. The recommendations on dataset filtering and metric evaluation particularly imply that significant improvements in computation load and evaluation integrity can be attained through strategic modifications.

Theoretically, this research extends the understanding of model biases induced by training datasets, implying a need for greater emphasis on dataset design and evaluation criteria in IL. It advocates for a nuanced consideration of expert style and data diversity, encouraging future research to deliberate on these previously underexamined influences.

Future Directions in Autonomous Driving

The study's findings bear substantial implications for the future of autonomous driving research. It beckons further exploration into the role of expert behavior modeling and dataset curation strategies, potentially inspiring a paradigm shift in how autonomous systems learn from simulated environments like CARLA. Future research could explore how varying styles and behaviors of the expert models could be optimized to enhance policy learning and generalization across diverse driving contexts.

In summary, this paper provides an important contribution to autonomous driving research, presenting robust insights into the biases embedded within end-to-end driving datasets and offering practical guidelines to address them. Its implications underscore the necessity for ongoing investigation into dataset design and evaluation practices to bolster the reliability and performance of autonomous driving models.