Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution

Abstract: Diffusion models are just at a tipping point for image super-resolution task. Nevertheless, it is not trivial to capitalize on diffusion models for video super-resolution which necessitates not only the preservation of visual appearance from low-resolution to high-resolution videos, but also the temporal consistency across video frames. In this paper, we propose a novel approach, pursuing Spatial Adaptation and Temporal Coherence (SATeCo), for video super-resolution. SATeCo pivots on learning spatial-temporal guidance from low-resolution videos to calibrate both latent-space high-resolution video denoising and pixel-space video reconstruction. Technically, SATeCo freezes all the parameters of the pre-trained UNet and VAE, and only optimizes two deliberately-designed spatial feature adaptation (SFA) and temporal feature alignment (TFA) modules, in the decoder of UNet and VAE. SFA modulates frame features via adaptively estimating affine parameters for each pixel, guaranteeing pixel-wise guidance for high-resolution frame synthesis. TFA delves into feature interaction within a 3D local window (tubelet) through self-attention, and executes cross-attention between tubelet and its low-resolution counterpart to guide temporal feature alignment. Extensive experiments conducted on the REDS4 and Vid4 datasets demonstrate the effectiveness of our approach.

• The paper proposes SATeCo, demonstrating that diffusion models with SFA and TFA can significantly enhance spatial fidelity and temporal coherence in video super-resolution.
• It integrates transformer-based upscaling with VAE and UNet modules to modulate pixel features and align frame information for consistent high-resolution synthesis.
• Extensive experiments on REDS4 and Vid4 show SATeCo achieves superior perceptual metrics compared to existing methods, bridging regression and diffusion approaches.

"Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution" Essay

Introduction

The research paper entitled "Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution" proposes a novel approach, SATeCo, aimed at capitalizing on diffusion models for video super-resolution (VSR). Unlike traditional methods, SATeCo focuses on addressing the dual challenges of maintaining spatial fidelity and ensuring temporal coherence across video frames. This is achieved by introducing spatial-temporal guidance that leverages low-resolution (LR) videos for high-resolution (HR) video synthesis through a combination of transformer-based upscaling, latent-space denoising, and pixel-space reconstruction strategies.

SATeCo Architecture Overview

The SATeCo architecture comprises several key components, depicted in Figure 1. Initially, an input LR video undergoes upscaling using a transformer-based video upscaler, producing a resolution-enhanced video that enters the Variational Autoencoder (VAE) pipeline. Within the VAE framework, spatial feature adaptation (SFA) and temporal feature alignment (TFA) modules are inserted into both the UNet and VAE decoders. These modules modulate pixel features and ensure temporal consistency through self-attention and cross-attention processes, as further detailed below.

Figure 1: An overview of our SATeCo architecture illustrating the workflow from LR video input to HR video output.

Spatial Feature Adaptation and Temporal Feature Alignment

The paper introduces SFA and TFA as critical modules for achieving high spatial fidelity and temporal coherence:

• Spatial Feature Adaptation: SFA ensures pixel-wise feature modulation by estimating affine parameters (scale and bias) derived from LR video latent features. This modulation is critical for enhancing spatial fidelity during the video denoising process within the latent space.
• Temporal Feature Alignment: TFA facilitates temporal coherence by leveraging tubelet-based self-attention and cross-attention mechanisms. This method aligns temporal features within a local 3D window (tubelet) across frames and calibrates HR features with their LR counterparts, addressing the challenge of frame inconsistency.

  Figure 2: An illustration of (c) spatial feature adaptation and (d) temporal feature alignment modules.

Experimental Evaluation

The paper conducts extensive experiments on the REDS4 and Vid4 datasets, demonstrating SATeCo's effectiveness. The results, summarized in Table 1, exhibit SATeCo's superiority in perception-based metrics, such as LPIPS and DISTS, when compared to competing methods like VRT and StableSR. Notably, the approach achieves comparable PSNR and SSIM scores alongside enhanced perceptual quality, affirming the model's ability to bridge the gap between traditional regression models and diffusion-based super-resolution models.

Figure 3: Six visual examples of video super-resolution results by different approaches on the REDS4 and Vid4 datasets.

Comparative Analysis and Model Evaluation

The integration of SFA and TFA within the SATeCo framework is found to significantly improve both spatial and temporal feature learning. As highlighted in Table 2, variants of the SATeCo model, incorporating only the SFA and TFA modules sequentially within UNet and VAE, show incremental performance gains over the baseline, culminating in superior overall results for the complete SATeCo implementation.

Figure 4: Video super-resolution results of two videos in the Vid4 dataset demonstrating temporal consistency.

Conclusion

SATeCo effectively advances the state of video super-resolution by leveraging spatial adaptation and temporal coherence mechanisms within diffusion models. By capitalizing on the pixel-wise learning of spatial guidance and temporal alignment from LR videos, SATeCo achieves high-quality HR video synthesis with improved spatial and temporal consistency. Future research directions may explore further optimization of the guiding mechanisms or integration with other generative models to enhance performance across diverse datasets.