Pixel-wise AdaLN Modulation for GANs

• The paper introduces pixel-wise AdaLN (SPN), a self-supervised scheme that learns a per-channel, two-region latent mask for pixel-adaptive modulation.
• It uses depthwise convolutions to convert learned masks into pixel-specific affine parameters, replacing conventional BN/LN in GAN generators.
• Integrating SPN into ResBlock architectures yields significant improvements in FID and IS, outperforming standard BN/cBN methods.

Pixel-wise AdaLN Modulation, referenced as Self Pixel-wise Normalization (SPN), is a normalization and modulation scheme for deep generative models, notably GANs, designed to enable pixel-adaptive affine transformations without external masks or segmentation maps. SPN learns a self-supervised, per-channel, two-region latent mask from the feature activations and uses this to generate distinct affine parameters for each pixel, enhancing image synthesis quality and spatial adaptability over traditional channel-wise or externally masked region-adaptive normalization. SPN can be directly inserted in place of BN or LN in existing ResBlock-based generator architectures and demonstrates significant gains in generative performance metrics such as FID and IS (Yeo et al., 2022).

1. Core Formulation of Pixel-wise AdaLN (SPN)

Given a 4D feature tensor $X \in \mathbb{R}^{B \times H \times W \times C}$ (batch, height, width, channel), classic channel-wise normalization (e.g., BN or LN) computes per-channel statistics:

• $\mu_c = \frac{1}{BHW}\sum_{b,p} x_{b,p,c}$
• $$\sigma_c = \sqrt{\frac{1}{BHW} \sum_{b,p} (x_{b,p,c}-\mu_c)^2 + \epsilon}$$

Each feature is normalized:

• $$\hat{x}_{b,p,c} = \frac{x_{b,p,c} - \mu_c}{\sigma_c}$$

In standard BN, a single pair $(\gamma_c, \beta_c)$ applies to each channel:

• $y_{b,p,c} = \gamma_c \hat{x}_{b,p,c} + \beta_c$

SPN generalizes this by producing pixel-specific affine parameters, $(\gamma_{b,p,c}, \beta_{b,p,c})$:

• $$y_{b,p,c} = \gamma_{b,p,c} \hat{x}_{b,p,c} + \beta_{b,p,c}$$

This approach offers full spatial specificity for normalization-induced modulation.

2. Self-Latent Mask Mechanism

Unlike spatially-adaptive normalization layers requiring externally provided masks (e.g., SPADE), SPN learns a two-region, foreground/background separation for each channel:

• For each $x(j) \in \mathbb{R}^{B \times H \times W}$:
  • $$m(j) = \sigma(\text{Conv}_j^{\text{mask}}[x(j)]) \in (0,1)^{B \times H \times W}$$
  • $\mu_c = \frac{1}{BHW}\sum_{b,p} x_{b,p,c}$0

No explicit mask regularization is employed. The adversarial loss and image formation objective naturally encourage $\mu_c = \frac{1}{BHW}\sum_{b,p} x_{b,p,c}$1 to form near-binary, semantically aligned (object vs. background) spatial masks. These two complementary masks partition the feature map space without access to any ground-truth segmentation.

3. Pixel-wise Modulation via Mask-based Convolution

SPN transforms $\mu_c = \frac{1}{BHW}\sum_{b,p} x_{b,p,c}$2 and $\mu_c = \frac{1}{BHW}\sum_{b,p} x_{b,p,c}$3 into modulation parameters through depthwise convolutions:

• $\mu_c = \frac{1}{BHW}\sum_{b,p} x_{b,p,c}$4
• $\mu_c = \frac{1}{BHW}\sum_{b,p} x_{b,p,c}$5

Here, each channel $\mu_c = \frac{1}{BHW}\sum_{b,p} x_{b,p,c}$6 possesses unique kernel pairs for each modulation path, producing fully pixel- and channel-specific affine transforms. The elementwise convolution ($\mu_c = \frac{1}{BHW}\sum_{b,p} x_{b,p,c}$7) ensures locality and expressiveness at every spatial position.

4. Integration in Generative Architectures

SPN modules replace all BN/cBN or LN layers in generator ResBlock structures above spatial resolution $\mu_c = \frac{1}{BHW}\sum_{b,p} x_{b,p,c}$8. For example, an SPN-ResBlock consists of:

• Conv $\mu_c = \frac{1}{BHW}\sum_{b,p} x_{b,p,c}$9 SPN $$\sigma_c = \sqrt{\frac{1}{BHW} \sum_{b,p} (x_{b,p,c}-\mu_c)^2 + \epsilon}$$0 ReLU $$\sigma_c = \sqrt{\frac{1}{BHW} \sum_{b,p} (x_{b,p,c}-\mu_c)^2 + \epsilon}$$1 Conv $$\sigma_c = \sqrt{\frac{1}{BHW} \sum_{b,p} (x_{b,p,c}-\mu_c)^2 + \epsilon}$$2 SPN $$\sigma_c = \sqrt{\frac{1}{BHW} \sum_{b,p} (x_{b,p,c}-\mu_c)^2 + \epsilon}$$3 (skip connection) $$\sigma_c = \sqrt{\frac{1}{BHW} \sum_{b,p} (x_{b,p,c}-\mu_c)^2 + \epsilon}$$4 out

Deployment details:

• For $$\sigma_c = \sqrt{\frac{1}{BHW} \sum_{b,p} (x_{b,p,c}-\mu_c)^2 + \epsilon}$$5 generators, three SPN layers inserted at $$\sigma_c = \sqrt{\frac{1}{BHW} \sum_{b,p} (x_{b,p,c}-\mu_c)^2 + \epsilon}$$6, $$\sigma_c = \sqrt{\frac{1}{BHW} \sum_{b,p} (x_{b,p,c}-\mu_c)^2 + \epsilon}$$7, $$\sigma_c = \sqrt{\frac{1}{BHW} \sum_{b,p} (x_{b,p,c}-\mu_c)^2 + \epsilon}$$8
• For $$\sigma_c = \sqrt{\frac{1}{BHW} \sum_{b,p} (x_{b,p,c}-\mu_c)^2 + \epsilon}$$9, five SPN-enabled stages

This design is demonstrated on SNGAN, BigGAN, and cGAN architectures and can be introduced without modification to architecture outside normalization replacement (Yeo et al., 2022).

5. Training Objectives and Hyperparameters

SPN-based GANs utilize standard adversarial objectives:

• Discriminator: $$\hat{x}_{b,p,c} = \frac{x_{b,p,c} - \mu_c}{\sigma_c}$$0
• Generator: $$\hat{x}_{b,p,c} = \frac{x_{b,p,c} - \mu_c}{\sigma_c}$$1

No auxiliary loss is required for the mask. Spectral normalization is applied to the discriminator and, in large-image settings, to the generator. Adam ($$\hat{x}_{b,p,c} = \frac{x_{b,p,c} - \mu_c}{\sigma_c}$$2) is used, with TTUR for high-resolution tasks.

6. Performance Gains and Comparative Analysis

SPN demonstrates consistent improvement in FID and IS across datasets and settings, as summarized below:

Architecture	Dataset/Setting	FID	IS
SNGAN+BN	CIFAR-10, unconditional GAN	13.46 ± 0.30	7.77
SNGAN+SPN	CIFAR-10, unconditional GAN	12.16 ± 0.16	7.93
SNGAN+cBN	CIFAR-10, class-conditional cGAN	10.21 ± 0.18	8.03
BigGAN	CIFAR-10, class-conditional cGAN	9.45 ± 0.15	8.03
Ours (cSPN)	CIFAR-10, class-conditional cGAN	7.72 ± 0.18	8.35
SNGAN+cBN	Tiny-ImageNet, 128×128 cGAN	35.42	20.52
BigGAN	Tiny-ImageNet, 128×128 cGAN	35.13	20.23
Ours (cSPN)	Tiny-ImageNet, 128×128 cGAN	28.31	23.35
SNGAN+BN	LSUN-church, 128×128 unconditional	≈8.07	–
SNGAN+SPN	LSUN-church, 128×128 unconditional	≈6.91	–

All metrics are obtained with identical architectures except for BN/cBN versus SPN substitution (Yeo et al., 2022).

7. Context, Limitations, and Distinctions from Other Methods

SPN (pixel-wise AdaLN) distinguishes itself from:

• Channel-wise BN/cBN: These provide single global $$\hat{x}_{b,p,c} = \frac{x_{b,p,c} - \mu_c}{\sigma_c}$$3 per channel, lacking spatial specificity.
• SPADE and other region-adaptive normalizations: These require externally supplied masks/segmentations.

SPN's self-latent mask is self-supervised and adapts per instance, enabling flexible, per-pixel modulation. The mechanism enables the network to specialize affine transforms for foreground versus background, typically converging to a binary region separation. A plausible implication is that this permits downstream convolutional blocks to focus more on refining shape and texture rather than encoding spatial layout.

SPN's universality and quantitative improvements have been established without any external mask supervision and solely by slot-in replacement in standard GANs, offering an alternative to both legacy channel-wise and externally masked normalization paradigms (Yeo et al., 2022).