UnfoldIR: Rethinking Deep Unfolding Network in Illumination Degradation Image Restoration

Abstract: Deep unfolding networks (DUNs) are widely employed in illumination degradation image restoration (IDIR) to merge the interpretability of model-based approaches with the generalization of learning-based methods. However, the performance of DUN-based methods remains considerably inferior to that of state-of-the-art IDIR solvers. Our investigation indicates that this limitation does not stem from structural shortcomings of DUNs but rather from the limited exploration of the unfolding structure, particularly for (1) constructing task-specific restoration models, (2) integrating advanced network architectures, and (3) designing DUN-specific loss functions. To address these issues, we propose a novel DUN-based method, UnfoldIR, for IDIR tasks. UnfoldIR first introduces a new IDIR model with dedicated regularization terms for smoothing illumination and enhancing texture. We unfold the iterative optimized solution of this model into a multistage network, with each stage comprising a reflectance-assisted illumination correction (RAIC) module and an illumination-guided reflectance enhancement (IGRE) module. RAIC employs a visual state space (VSS) to extract non-local features, enforcing illumination smoothness, while IGRE introduces a frequency-aware VSS to globally align similar textures, enabling mildly degraded regions to guide the enhancement of details in more severely degraded areas. This suppresses noise while enhancing details. Furthermore, given the multistage structure, we propose an inter-stage information consistent loss to maintain network stability in the final stages. This loss contributes to structural preservation and sustains the model's performance even in unsupervised settings. Experiments verify our effectiveness across 5 IDIR tasks and 3 downstream problems.

UnfoldIR: Advancements in Illumination Degradation Image Restoration via Deep Unfolding Networks

The paper "UnfoldIR: Rethinking Deep Unfolding Network in Illumination Degradation Image Restoration" presents a novel methodology for addressing illumination degradation in image restoration. The authors explore the integration of model-based and learning-based approaches through Deep Unfolding Networks (DUNs), introducing a sophisticated architecture known as UnfoldIR designed specifically for the Illumination Degradation Image Restoration (IDIR) tasks.

Overview of UnfoldIR Methodology

UnfoldIR capitalizes on the strength of DUNs by introducing an innovative IDIR model that incorporates dedicated regularization terms for illumination smoothness and texture enhancement. At its core, UnfoldIR deploys a multistage network architecture that unfolds iterative solutions into stages, each comprising two primary modules: Reflectance-Assisted Illumination Correction (RAIC) and Illumination-Guided Reflectance Enhancement (IGRE). RAIC employs Visual State Space (VSS) modules to extract non-local features for enforcing illumination smoothness, while IGRE introduces a frequency-aware extension of the VSS to align similar textures globally, providing guidance from mildly degraded to severely degraded regions.

Key Contributions

1. IDIR Model with Explicit Regularization Terms: The IDIR model proposed in UnfoldIR is inspired by Retinex theory and introduces explicit constraints for illumination and reflectance components, aiming to optimize texture preservation and suppress imaging noise effectively.
2. Multistage Architecture with RAIC and IGRE Modules: Each stage is meticulously constructed, leveraging a combination of VSS and frequency-aware modifications to refine illumination and texture details progressively.
3. Inter-Stage Information Consistent (ISIC) Loss: The ISIC loss uniquely serves the DUN framework by ensuring network stability, enhancing structural details while preventing distortion, especially valuable in unsupervised settings.
4. Extensive Evaluation Across Multiple Tasks: UnfoldIR’s effectiveness was demonstrated through comprehensive experiments on five IDIR tasks and three downstream problems, emphasizing improvements in performance metrics like PSNR, SSIM, FID, and BIQE.

Experimental Insights

The experimental results indicate that UnfoldIR markedly surpasses existing state-of-the-art methods across various datasets, displaying significant performance improvements in both efficiency and accuracy. Notably, numerical results reported substantial gains in PSNR and SSIM, corroborating UnfoldIR's efficacy in handling complex illumination-degraded scenarios. Furthermore, the architectural enhancements, particularly the VSS and ISIC loss components, concretely contributed to its competitive advantage.

Implications and Future Directions

UnfoldIR encapsulates a pivotal advancement in the broader field of image restoration by demonstrating the practical viability of integrating unfolding and generative paradigms. From a theoretical perspective, the exploration of the unfolding structure paves the way for future endeavors to refine this synergy further, potentially expanding into other modalities and enhancing multi-modal image processing tasks. Given the promising results, future research may explore incorporating generative diffusion models with UnfoldIR's framework to cultivate improved perceptual fidelity and address its limitations in recovering intricate texture details.

The paper’s contributions significantly broaden the applicability of DUNs, heralding novel avenues in restoration methodologies and fostering dialogue on the potential convergence of model-based principles with learning-based adaptability in image restoration disciplines.