Image Super-Resolution with Text Prompt Diffusion

Abstract: Image super-resolution (SR) methods typically model degradation to improve reconstruction accuracy in complex and unknown degradation scenarios. However, extracting degradation information from low-resolution images is challenging, which limits the model performance. To boost image SR performance, one feasible approach is to introduce additional priors. Inspired by advancements in multi-modal methods and text prompt image processing, we introduce text prompts to image SR to provide degradation priors. Specifically, we first design a text-image generation pipeline to integrate text into the SR dataset through the text degradation representation and degradation model. By adopting a discrete design, the text representation is flexible and user-friendly. Meanwhile, we propose the PromptSR to realize the text prompt SR. The PromptSR leverages the latest multi-modal LLM (MLLM) to generate prompts from low-resolution images. It also utilizes the pre-trained LLM (e.g., T5 or CLIP) to enhance text comprehension. We train the PromptSR on the text-image dataset. Extensive experiments indicate that introducing text prompts into SR, yields impressive results on both synthetic and real-world images. Code: https://github.com/zhengchen1999/PromptSR.

• The paper proposes PromptSR, a novel image super-resolution method that integrates text prompts describing image degradations to guide a diffusion model for enhanced restoration.
• The authors created a text-augmented dataset using a degradation model and text representations, and developed PromptSR using pre-trained language models and a diffusion framework trained on this data.
• Empirical tests show PromptSR outperforms state-of-the-art methods on benchmarks, demonstrating that textual degradation priors significantly improve generalization and performance, especially in complex scenarios.

Overview of "Image Super-Resolution with Text Prompt Diffusion"

The paper "Image Super-Resolution with Text Prompt Diffusion" presents an innovative approach to single image super-resolution (SR) by introducing text prompts to leverage degradation priors. Acknowledging the limitations of conventional SR methods that rely primarily on low-resolution image degradation modeling, this research explores the potential of integrating textual information into the super-resolution domain. The authors aim to enhance model performance by providing degradation knowledge through text prompts that capitalize on advances in multi-modal learning and pre-trained LLMs.

Methodology

The research introduces a two-pronged strategy involving a text-image generation pipeline and a specialized model termed PromptSR:

1. Text-Image Generation Pipeline: The authors design a pipeline to generate a text-augmented super-resolution dataset that incorporates degradation descriptions. The core components of this pipeline are:
   • Degradation Model: A comprehensive yet straightforward model is used to simulate common image degradations such as blurring, resizing, noise addition, and compression. This methodology allows the generation of low-resolution images from high-resolution counterparts.
   • Text Representation: Degradation characteristics are faithfully captured using text prompts. By adopting a discretization and binning approach, text prompts offer abstract but comprehensive descriptions of degradation, enhancing flexibility and user-friendliness. They circumvent the complexities of precise quantitative descriptions while maintaining sufficient granularity.
2. PromptSR Model: To handle the text prompt SR, PromptSR employs the capabilities of pre-trained LLMs like T5 or CLIP to process degradation representations effectively:
   • This model utilizes a diffusion model framework, incorporating text information to guide the SR task. Conditional on both the low-resolution image and textual embeddings, PromptSR enhances image restoration while being trained on the generated text-image dataset.

Results and Implications

Empirical evaluations reveal that integrating text prompts that describe degradations can significantly bolster SR performance on synthetic and real-world datasets. The introduction of textual degradation descriptions allows for improved generalization and performance, especially in complex and uncertain degradation scenarios. Quantitative assessments on benchmarks like Urban100, Manga109, and RealSR affirm the efficacy of PromptSR, demonstrating notable improvements in LPIPS, SSIM, and visual quality metrics when compared with state-of-the-art single-modal SR models.

The study's exploration into multi-modal approaches for SR opens avenues for further research in leveraging LLMs across other image restoration tasks. By providing degradation priors, text prompts potentially reduce the reliance on exhaustive HR-LR pair generation, promising more efficient dataset creation methodologies. Furthermore, by harnessing powerful pre-trained network models, future enhancements may focus on refining text prompt representations and model architectures to balance computational efficiency and restoration fidelity.

In conclusion, this study enriches the field of image super-resolution by presenting a novel fusion of text and image modalities. The implications extend beyond SR towards broader applications in image manipulation and restoration, heralding new opportunities where textual cues significantly contribute to visual data enhancement and understanding.