Training Strategies for Vision Transformers for Object Detection

Abstract: Vision-based Transformer have shown huge application in the perception module of autonomous driving in terms of predicting accurate 3D bounding boxes, owing to their strong capability in modeling long-range dependencies between the visual features. However Transformers, initially designed for LLMs, have mostly focused on the performance accuracy, and not so much on the inference-time budget. For a safety critical system like autonomous driving, real-time inference at the on-board compute is an absolute necessity. This keeps our object detection algorithm under a very tight run-time budget. In this paper, we evaluated a variety of strategies to optimize on the inference-time of vision transformers based object detection methods keeping a close-watch on any performance variations. Our chosen metric for these strategies is accuracy-runtime joint optimization. Moreover, for actual inference-time analysis we profile our strategies with float32 and float16 precision with TensorRT module. This is the most common format used by the industry for deployment of their Machine Learning networks on the edge devices. We showed that our strategies are able to improve inference-time by 63% at the cost of performance drop of mere 3% for our problem-statement defined in evaluation section. These strategies brings down Vision Transformers detectors inference-time even less than traditional single-image based CNN detectors like FCOS. We recommend practitioners use these techniques to deploy Transformers based hefty multi-view networks on a budge-constrained robotic platform.

• The paper provides a comprehensive analysis of how training strategies optimize ViT-based object detectors with minimal accuracy loss while significantly reducing inference time.
• It demonstrates that architectural downscaling, including reduced image resolution and tailored decoder settings, can achieve up to a 63% runtime reduction with minor performance trade-offs.
• Deployment-oriented techniques such as reduced precision, one-cycle learning rates, and post-processing adjustments further enhance real-time performance for edge-based autonomous driving.

Training Strategies for Vision Transformers for Object Detection: An Expert Review

Introduction and Context

The paper "Training Strategies for Vision Transformers for Object Detection" (2304.02186) systematically investigates the optimization of vision transformer (ViT)-based object detectors for autonomous driving under deployment constraints, prioritizing the joint optimization of detection performance (mAP, max-F1) and inference-time. Unlike most prior work that centers on architectural advances and leaderboard performance, this work rigorously dissects the effects of distinct training manipulations, architectural down-scaling, and deployment-oriented precision strategies, directly addressing the real-world requirements of edge deployment, such as on-board compute in autonomous vehicles.

A canonical architecture in this domain, where a CNN backbone and FPN feed scene representations to a transformer-based detection head, is illustrated in (Figure 1).

Figure 1: Architecture of a multi-view vision-based detector with CNN backbone, FPN, and transformer detection head; inputs are multi-camera images and calibration, outputs are 3D bounding boxes.

Object Detection Under Real-World Constraints

The deployment context for autonomous driving mandates fast, robust perception across the entire $360^\circ$ scene, typically constructed from multiple cameras. Classical two-stage detectors (e.g., Faster R-CNN, Mask R-CNN) are outperformed in terms of latency by single-stage detectors (e.g., YOLO, FCOS), but both CNN approaches are limited by restricted receptive field and weak modeling of long-range dependencies.

The emergence of ViTs and DETR-based approaches enables direct set-based prediction with powerful self-attention-driven context modeling, essential for object detection in cluttered, occluded, or heavily multiplexed urban scenes. However, as shown in this study, the computational overhead of naïve transformer deployment in these pipelines can severely restrict practical utility.

The task involves 3D object localization from synchronized camera images, against HD map and LiDAR overlays for ground truth, in a complex urban driving domain, as depicted in (Figure 2).

Figure 2: The multi-view object detection problem: input is 8-camera surround imagery; output is 3D BEV box detection, aligned with HD map and LiDAR signal.

Profiling Computational Bottlenecks

A breakdown of profiling reveals that the substantial bulk of inference cost arises from the CNN backbone and FPN stages rather than the transformer head. Empirical measurements demonstrate that network-wide inference time reductions follow quadratically from down-scaling the input resolution, while parameters associated with the detection head (embedding size, queries, number of decoders) tune the speed-accuracy tradeoff at the transformer stage.

Key findings are:

• Reducing input image dimensions by 40% yields a 52.5% reduction in inference time, with minimal $<1.5\%$ AP reduction (vehicles, pedestrians), maintaining competitive detection metrics.
• Cropping 50 least-informative pixels from the top of images (sky, irrelevant background) gives an additional 10% runtime reduction, with negligible impact on detection accuracy.

Such strategies are robust to deployment hardware and not simply parameter count proxies (e.g., MACs do not fully predict actual runtime on target GPUs).

Network Downscaling Experiments

A broad grid search explores further model contractions:

• Decoder Layer Depth: Six decoder layers yield optimal speed-accuracy balance. Fewer decoders result in marked precision degradation, while adding further layers provides diminishing returns but increases latency.
• Embedding Dimension: Reducing query and FPN channel embedding from 256 to 128 produces a 9.5% runtime gain with no performance loss, likely due to hardware alignment and sufficient representation capacity.
• Query Count: Empirically, setting the transformer query count to the approximate expected object count per 360° scene, plus buffer, avoids excessive redundant predictions. Lowering from 900 to 400 queries reduces compute by 5% and can actually improve precision due to sparser, cleaner outputs.
• Training Schedule: One-cycle learning rate scheduling, with wide swings from low initial/final to high mid-epoch values, accelerates convergence and improves generalization, outperforming fixed or narrow range schedules.

These results indicate standard transformer detection heads are significantly over-provisioned for practical scene complexity, and judicious contraction is possible with modest or no accuracy penalty.

Precision, Format, and Runtime Profiling

A direct comparison between PyTorch and TensorRT reveals that deployment-aware optimization with reduced precision (float16) confers dramatic runtime benefits. On a T4 Nvidia GPU, float16 models in TensorRT execute at 62 ms per frame—less than half the float32 PyTorch runtime—resulting in overall 63% improvement in end-to-end latency with negligible AP loss (~3% at worst).

This confirms that edge deployment necessitates tight integration of training, architecture, and hardware-aware quantization strategies, which have typically been siloed in prior work.

Advanced Post-Processing and Empirical Observations

Critical post-training manipulations include:

• Application of Non-Maximum Suppression (NMS), even in transformer-based detectors that theoretically should not require it, yields notable improvements for large, overlapping classes (vehicles), addressing the imperfect set loss alignment with real distributions.
• Empirical tuning of the number of top-K retained boxes (e.g., set to 90% of number of queries) optimizes the final candidate set, reducing false positives at negligible cost.

Implications and Theoretical Synthesis

The research challenges the prevailing focus on ever-larger vision transformer architectures for object detection in autonomous driving by demonstrating that targeted downscaling, deployment-time quantization, and systematic training strategies can vastly reduce inference requirements with minimal impact on accuracy. These findings are antithetical to brute-force leaderboard chasing typical in the field, and instead advocate for a deployment-centric paradigm, where speed-accuracy tradeoffs are empirically traced and optimized.

Suggested future research directions, such as neural architecture search (NAS) tailored for ViTs and transformer-specific pruning schemes, are anticipated to further close the gap toward real-time, energy-efficient, high-fidelity perception on resource-constrained robotic platforms.

Conclusion

This paper provides a comprehensive, empirically validated set of strategies for maximizing the inference efficiency of vision transformer-based object detectors targeted at real-time autonomous driving scenarios. The demonstrated ability to effect a 63% runtime reduction with only marginal accuracy loss sets new expectations for practical object detection pipeline design, establishing a valuable reference for both research and industrial deployment. The broader implication is a shift toward environmentally sustainable, cost-effective deep perception systems, with theoretical relevance across domains wherever resource constraints collide with high performance needs.