Omni Aggregation Networks for Lightweight Image Super-Resolution

Abstract: While lightweight ViT framework has made tremendous progress in image super-resolution, its uni-dimensional self-attention modeling, as well as homogeneous aggregation scheme, limit its effective receptive field (ERF) to include more comprehensive interactions from both spatial and channel dimensions. To tackle these drawbacks, this work proposes two enhanced components under a new Omni-SR architecture. First, an Omni Self-Attention (OSA) block is proposed based on dense interaction principle, which can simultaneously model pixel-interaction from both spatial and channel dimensions, mining the potential correlations across omni-axis (i.e., spatial and channel). Coupling with mainstream window partitioning strategies, OSA can achieve superior performance with compelling computational budgets. Second, a multi-scale interaction scheme is proposed to mitigate sub-optimal ERF (i.e., premature saturation) in shallow models, which facilitates local propagation and meso-/global-scale interactions, rendering an omni-scale aggregation building block. Extensive experiments demonstrate that Omni-SR achieves record-high performance on lightweight super-resolution benchmarks (e.g., 26.95 dB@Urban100 $\times 4$ with only 792K parameters). Our code is available at \url{https://github.com/Francis0625/Omni-SR}.

• The paper introduces Omni-SR, which enhances lightweight ViT models by integrating omni self-attention and omni-scale aggregation to overcome uni-dimensional limitations.
• The OSA block simultaneously processes spatial and channel dimensions, enabling richer pixel interactions and more effective contextual modeling.
• The OSAG scheme leverages local, meso, and global context for robust feature aggregation, achieving a PSNR of 26.95 dB on Urban100 with only 792K parameters.

Omni Aggregation Networks for Lightweight Image Super-Resolution

The paper "Omni Aggregation Networks for Lightweight Image Super-Resolution" introduces an innovative architecture, Omni-SR, designed to enhance the performance of lightweight vision transformer (ViT)-based models specifically tailored for image super-resolution (SR) tasks. The core contributions of the work revolve around addressing the limitations of uni-dimensional self-attention mechanisms and homogeneous aggregation strategies in current models.

The paper identifies two major challenges in lightweight ViT-based models: limited effective receptive fields (ERF) due to uni-dimensional attention and insufficient multi-scale feature aggregation. To surmount these difficulties, the authors propose two principal components: the Omni Self-Attention (OSA) block and the Omni-Scale Aggregation Group (OSAG).

Omni Self-Attention (OSA) Block

The OSA block is a novel self-attention mechanism that simultaneously considers both spatial and channel dimensions, thus working across omni-axes. This dual-axis approach enables more comprehensive pixel interactions, enhancing the ViT's capability to model complex dependencies in the input data while adhering to computational constraints through window partitioning strategies. The dense interaction facilitated by the OSA block allows for more sophisticated contextual information extraction compared to traditional spatial-only or channel-only self-attention mechanisms, substantially improving the attention's expressive power.

Omni-Scale Aggregation Group (OSAG)

The OSAG addresses issues of premature saturation by introducing a multi-scale interaction scheme that captures local, meso, and global context simultaneously. This hierarchical feature aggregation improves the effective receptive field by leveraging local convolution for detail capture, meso self-attention for mid-scale interaction, and global self-attention for broader contextual understanding. This structured approach ensures a more robust information propagation pathway, amplifying the model's ability to handle intricate SR tasks effectively.

Experimental Insights

The experimental results demonstrate that the Omni-SR framework achieves state-of-the-art performance across several benchmarks, notably achieving a PSNR of 26.95 dB on the Urban100 dataset with only 792K parameters. Comparative analyses with existing methods highlight the superiority of the proposed approach, especially in terms of optimization efficiency and information encoding capabilities.

Implications and Future Directions

The implications of this research extend both theoretically and practically. Theoretically, it underscores the significance of a multi-dimensional approach in self-attention mechanisms and the advantages of an omni-scale aggregation strategy, setting a precedent for future exploration in transformer architectures for SR.

Practically, the lightweight nature coupled with high performance of Omni-SR makes it particularly suitable for edge devices and applications requiring real-time processing. The ability to deliver high-quality image restoration with fewer computational resources aligns well with practical deployment scenarios in diverse fields such as medical imaging, satellite image processing, and mobile photography.

Furthermore, the findings prompt several avenues for further research, including exploring the adaptability of the proposed components in broader low-level vision tasks and optimizing the balance between complexity and performance in ultra-lightweight deployment scenarios. There are also prospects to investigate the adaptability of the OSA and OSAG components to other domain-specific architectures beyond image super-resolution.

In conclusion, the paper makes substantial contributions to the domain of lightweight image super-resolution, providing an innovative architecture that efficiently leverages both spatial and channel interactions and effectively aggregates multi-scale features. This work enriches the current understanding and design of efficient vision transformers, paving the way for further advancements in the field.