LowFormer: Hardware Efficient Design for Convolutional Transformer Backbones

Abstract: Research in efficient vision backbones is evolving into models that are a mixture of convolutions and transformer blocks. A smart combination of both, architecture-wise and component-wise is mandatory to excel in the speedaccuracy trade-off. Most publications focus on maximizing accuracy and utilize MACs (multiply accumulate operations) as an efficiency metric. The latter however often do not measure accurately how fast a model actually is due to factors like memory access cost and degree of parallelism. We analyzed common modules and architectural design choices for backbones not in terms of MACs, but rather in actual throughput and latency, as the combination of the latter two is a better representation of the efficiency of models in real applications. We applied the conclusions taken from that analysis to create a recipe for increasing hardware-efficiency in macro design. Additionally we introduce a simple slimmed-down version of MultiHead Self-Attention, that aligns with our analysis. We combine both macro and micro design to create a new family of hardware-efficient backbone networks called LowFormer. LowFormer achieves a remarkable speedup in terms of throughput and latency, while achieving similar or better accuracy than current state-of-the-art efficient backbones. In order to prove the generalizability of our hardware-efficient design, we evaluate our method on GPU, mobile GPU and ARM CPU. We further show that the downstream tasks object detection and semantic segmentation profit from our hardware-efficient architecture. Code and models are available at https://github.com/ altair199797/LowFormer.

• The paper proposes a hardware-efficient architecture that strategically fuses convolutional layers with transformer blocks to enhance throughput and accuracy.
• The paper introduces a lightweight multi-head self-attention mechanism that compresses channels and down-samples feature maps to reduce computational cost.
• The paper demonstrates superior performance across diverse hardware platforms, achieving high accuracies and faster processing on vision tasks.

An Overview of LowFormer: Hardware Efficient Design for Convolutional Transformer Backbones

The paper "LowFormer: Hardware Efficient Design for Convolutional Transformer Backbones" by Nottebaum et al. presents a novel architecture named LowFormer, which melds convolutional layers with transformer blocks to optimize the speed-accuracy trade-off in neural network design for vision tasks. Unlike conventional approaches that measure efficiency via Multiply-Accumulate Operations (MACs), the authors assess model performance considering actual throughput and latency, providing a more realistic benchmark for real-world applications.

Key Contributions

1. Macro Design for Hardware Efficiency: The authors conduct a detailed analysis of common modules and architectural designs in terms of latency and throughput. They identify inefficiencies in depthwise convolutions and higher resolution operations, advocating for a hardware-conscious design that minimizes these elements.
2. Micro Design - Lightweight Multi-Head Self-Attention (MHSA): The paper introduces a simplified, more hardware-efficient adaptation of the traditional MHSA. This adaptation compresses the channel dimensions and down-samples the feature maps before the Scaled Dot-Product Attention (SDA), significantly reducing computational demands without compromising accuracy.
3. Benchmarking Across Hardware: The robustness of LowFormer is validated across different hardware platforms, including GPU, mobile GPU, and ARM CPU. The results indicate substantial improvements in efficiency compared to state-of-the-art models, showcasing the model's generalizability to various deployment environments.

Analysis and Results

The paper provides strong empirical evidence supporting the proposed adaptations:

• Execution Time Analysis shows that standard convolutions outperform depthwise convolutions in practical hardware execution, despite having more MACs. For example, an unfused MBConv with higher channel dimensions but lower resolution proves faster than its depthwise counterpart.
• The impact of resolution on execution time is extensively analyzed, demonstrating that higher resolutions disproportionately slow down processing compared to higher channel dimensions. This insight is critical in designing the LowFormer architecture to operate efficiently at lower resolutions.
• LowFormer Architecture: The LowFormer family of models (B0-B3) is crafted using insights from the analysis, focusing computational complexity in later stages where resolutions are reduced. This strategic allocation leads to models that are both fast and accurate.

Performance Metrics

• Classification on ImageNet-1K: LowFormer variants consistently achieve high top-1 accuracy while maintaining superior throughput. For instance, LowFormer-B3 achieves a top-1 accuracy of 83.6% with a throughput of 6098 images/s, significantly outperforming comparable models like FAT-B3 and FastViT-SA36 both in terms of speed and accuracy.
• Downstream Task Performance: In object detection and semantic segmentation tasks, LowFormer backbones integrated into frameworks like RetinaNet and Semantic FPN exhibit substantial improvements in mean Average Precision (mAP) and mean Intersection over Union (mIoU), respectively. For instance, LowFormer-B3 achieves a 43.1% AP in COCO object detection and 44.6% mIoU in ADE20K semantic segmentation, outperforming several high-throughput models.

Implications and Future Directions

The innovations presented in LowFormer hold significant practical implications for deploying efficient and accurate neural networks on resource-constrained devices like smartphones and edge devices. By emphasizing throughput and latency over MACs, the authors provide a more practical measure of model efficiency, aligned with real-world deployment scenarios.

Conclusion

The paper convincingly demonstrates that the strategic combination of hardware-conscious macro and micro designs can yield models that excel in the speed-accuracy trade-off. By addressing inefficiencies in commonly used convolutions and attention mechanisms, LowFormer sets a benchmark for future research in efficient neural network architectures. Future developments in this domain may further explore the balance between computational efficiency and scalability, potentially incorporating emerging hardware optimizations and more refined design strategies.