GTFMN: Guided Texture and Feature Modulation

• GTFMN is a network architecture that uses dual- or multi-stream designs to combine external guidance (e.g., illumination, semantic maps) with feature restoration for adaptive image enhancement.
• Its explicit guidance mechanism, including the IGM block, modulates intermediate features using per-pixel cues to target underexposed and degraded regions effectively.
• Quantitative benchmarks and ablation studies demonstrate GTFMN’s superiority in preserving fine details and reducing noise compared to conventional single-branch restoration models.

The Guided Texture and Feature Modulation Network (GTFMN) defines a family of architectures for image restoration and enhancement that leverage explicit control over feature refinement based on external, spatially-varying guidance. Across modern instantiations, GTFMN is a dual- or multi-stream design, combining specialized streams for “guidance” signals (such as illumination, semantic maps, or high-level structure cues) with a main feature extraction and reconstruction stream. Inter-stream coupling is performed through a dedicated modulation mechanism operating on intermediate features, enabling the system to direct restoration efforts adaptively according to variable scene attributes.

1. Core Principles and Motivation

GTFMN addresses compounded degradation scenarios where conventional end-to-end super-resolution or enhancement networks often fail—especially in settings combining low spatial resolution with spatially uneven image quality due to illumination variation or challenging environmental conditions. In low-light image super-resolution (LLSR), the fundamental challenge is to recover high-frequency texture simultaneously with correcting for severe, sometimes non-uniform, underexposure. Naïve single-branch models tend to amplify both signal and noise indiscriminately, often degrading already well-lit regions when attempting to compensate for darker patches. GTFMN decouples the inherently ill-posed problem into two sub-tasks:

• Guidance stream: Predicts per-pixel or per-region attribute maps (e.g., illumination, semantic class, structural saliency) that inform the spatial pattern of degradation.
• Feature stream: Restores high-frequency detail and color but does so with its features modulated (or reweighted) under the guidance signals. This division of labor allows each network branch to specialize—guidance branches focus on low-frequency, context-driven understanding, while feature restoration branches optimize for fine details. The result is a pipeline capable of spatially adaptive enhancement, intensifying restoration only where necessary and preserving fidelity elsewhere (Huang et al., 27 Jan 2026, Wang et al., 2022, Wang et al., 2018).

2. Canonical Architectures and Modulation Mechanisms

Implementations of GTFMN span several domains, including LLSR and underwater image enhancement, but share the following structural motifs:

Illumination Guided Feature Modulation (LLSR)

In (Huang et al., 27 Jan 2026), the GTFMN is constructed as parallel Illumination and Texture Streams:

• Illumination Stream: Encoder-decoder structure produces an illumination map $M(x,y) \in [0,1]^{H \times W}$, recombined with a global exposure scalar to maintain photometric consistency. The spatially resolved map is used downstream to guide feature modulation.
• Texture Stream: Processes the low-light, low-resolution image through a series of blocks—each modulated by the guidance map.

Illumination-Guided Modulation (IGM) Block

Within the texture stream, the IGM block performs the following sequence:

1. LayerNorm of features $\hat F = \mathrm{LayerNorm}(F_{i-1})$.
2. Self-attention: Attentional activity derived from $\hat F$.
3. Guided attention: Guidance map $M$ is downsampled, passed through 1×1 convolutions with ReLU non-linearities to obtain (per-channel, per-location) gain ($\gamma_{\mathrm{guide}}$) and bias ($\beta_{\mathrm{guide}}$).
4. Attention fusion: Additively combines self- and guided-attention.
5. Feature modulation: Applies affine modulation:

$$F_{\mathrm{mod}} = \gamma_{\mathrm{guide}} \odot \hat F + \beta_{\mathrm{guide}}$$

1. Feed-forward and residual connection:

$F_i = F_{i-1} + \mathrm{FFN}(F_{\mathrm{mod}})$

This mechanism enables GTFMN to locally amplify features in underexposed regions while applying minimal modification to bright regions, ensuring spatially adaptive behavior.

Texture-Structure Modulation (Underwater Enhancement)

In underwater restoration (Wang et al., 2022), guidance is drawn from a pretrained VGG16 that produces shallow “texture” features and deeper “structure” features for each input image.

• Contextual Feature Refinement Module (CFRM): Operates in parallel with multiple convolutional kernel sizes to capture multi-scale context in both texture and structure streams.
• Feature Dominative Network (FDNet): Produces channel-wise modulation gates through global pooling and fully connected layers, guiding the UNet-like decoder at every upsampling stage:

$$f_{\mathrm{ste}}^i(p,q,c) = f_t^i(p,q,c) \cdot w_t^i(c)$$

where $w_t^i$ is computed from semantic-augmented features.

Semantic-Guided Feature Modulation in SFT (Wang et al., 2018)

Semantically-aware SR settings use spatial priors (e.g., segmentation maps) to drive affine modulation:

$$\mathrm{SFT}(F\mid \gamma,\beta) = \gamma \odot F + \beta$$

The affine parameters are generated from the segmentation probability maps via a learned condition network.

Variant	Guidance Input	Modulation Type	Application Domain
(Huang et al., 27 Jan 2026)	Illumination map	Per-block $\gamma, \beta$	Low-light SR/enhancement
(Wang et al., 2022)	Texture, structure (VGG16)	Channel gates	Underwater image enhancement
(Wang et al., 2018)	Semantic segmentation	Affine modulation	Semantic class-aware super-res

3. Training Protocols and Datasets

Training protocols are tailored to the challenges of the degradation type:

• LLSR GTFMN (Huang et al., 27 Jan 2026):
  • Loss: L1 pixel reconstruction ($$L_{\mathrm{rec}} = \|I_{\mathrm{SR}} - I_{\mathrm{HR}}\|_1$$), no adversarial or perceptual losses.
  • Optimization: Adam (β₁=0.9, β₂=0.999, ε=10⁻⁸), initial lr=2×10⁻⁴, halved every 50 epochs, batch size=16, 100 epochs total.
  • Synthetic low-light LR pairs via gamma correction (γ∈[0.4,0.6]) and bicubic downsampling ($s\in\{2,4\}$).
  • Benchmarks: OmniTrain (1600 HR images), OmniNormal5, OmniNormal15 (natural indoor/outdoor scenes).
• Underwater GTFMN (Wang et al., 2022):
  • Loss: Weighted combination of multi-scale SSIM and L1 ($\lambda = 0.8$).
  • Data: VGG16 pretrained on ImageNet for semantic extraction, underwater datasets with 224×224 crops, batch size=8, 100k iterations.
• SFT-based methods (Wang et al., 2018):
  • Loss: Perceptual (VGG), adversarial, auxiliary semantic classification.
  • Datasets: Outdoor images with 7 semantic classes, segmentation network fixed during SR network training.

4. Quantitative and Qualitative Results

GTFMN architectures consistently demonstrate state-of-the-art performance on benchmarks relevant to their domains:

• On OmniNormal5 ((Huang et al., 27 Jan 2026), LLSR), GTFMN achieves for ×2 SR: PSNR 38.34 dB versus ESRGAN’s 38.14 dB; SSIM 0.9833 versus 0.9830. For ×4 SR: PSNR 31.14 dB versus ESRGAN’s 30.05 dB, SSIM 0.9303 vs 0.9149. Comparable or higher gains are observed with larger test sets (OmniNormal15).
• Qualitatively, GTFMN avoids over-brightening and noise amplification in dark patches, producing legible text and signage, and restoring authentic scene details not achieved by baselines.
• Ablation in (Huang et al., 27 Jan 2026) confirms explicit illumination guidance is vital: removing the illumination stream drops PSNR by ≈0.1 dB and SSIM by ≈0.003.
• (Wang et al., 2022) shows “semantic-aware” GTFMN outperforms state-of-the-art underwater enhancement algorithms by large margins, notably in robustness to unseen conditions and suitability as a pre-processing step for downstream marine vision tasks.

5. Comparative Methodologies and Theoretical Connections

GTFMN relates directly to the broader family of spatially-adaptive feature modulation techniques:

• SFT layers (Wang et al., 2018): Demonstrate that conditional per-channel, per-location affine modulation improves texture faithfulness by incorporating semantic priors. This approach is parameter-efficient, as multiple blocks share a “condition network” output.
• Semantic-aware modulation in (Wang et al., 2022) decomposes features using pretrained encoders, with multi-path refinement and channel-wise modulation, enabling the model to distinguish texture from structure and adapt enhancement accordingly.
• IGM blocks (Huang et al., 27 Jan 2026) generalize spatial feature transform mechanisms by allowing modulation based on estimated illumination, rather than supervised semantic priors.

A plausible implication is that the architectural flexibility of GTFMN allows its adaptation to any task where spatial/non-spatial priors can be extracted and employed to guide restoration, extending well beyond traditional SR or enhancement pipelines.

6. Limitations, Ablations, and Prospects

Identified limitations and potential extensions include:

• Synthetic data constraints: LLSR results in (Huang et al., 27 Jan 2026) rely on gamma/bicubic pairs; real noise models remain unexplored.
• Efficiency: While parameter count is low (8.78M in (Huang et al., 27 Jan 2026)), further reductions and lightweight variants are proposed for on-device deployment.
• Generality: While current methods use fixed feature extractors or illumination heuristics, future variants could explore trainable guidance streams or apply GTFMN to video and temporally stable restoration.
• Ablation studies demonstrate the critical role of explicit guidance: reductions in PSNR/SSIM and qualitative artifacts when guidance streams or modulation are ablated confirm their necessity for spatial adaptivity.

7. Summary and Outlook

Guided Texture and Feature Modulation Networks represent a shift in enhancement and restoration by combining explicit, learned guidance streams with adaptive, pointwise feature modulation. GTFMN and its variants extend spatially-varying affine transformation concepts (as pioneered in SFT (Wang et al., 2018)) to address compounded degradations including uneven illumination and underwater image distortion. Extensive empirical evidence confirms quantitative and qualitative superiority over prior art, with robustness to diverse and challenging environments. Prospective directions include integration of real-world noise models, cross-modal guidance, and application to temporal sequences and other image-to-image translation domains (Huang et al., 27 Jan 2026, Wang et al., 2022, Wang et al., 2018).