Relativistic Average GAN (RaGAN)

• Relativistic Average GAN (RaGAN) is defined as a GAN variant that replaces absolute discrimination with relative probability estimates by comparing each sample against the average of the opposing class.
• It improves convergence and mitigates issues like mode collapse by enforcing batchwise coupling, leading to superior gradient dynamics and lower FID scores in applications such as super-resolution and signal denoising.
• Practical implementations show that RaGAN achieves robust performance with minimal computational overhead, leveraging dual-discriminator setups and optimized training heuristics across various domains.

Relativistic Average GAN (RaGAN) is a variant of generative adversarial networks that introduces a relativistic approach to the discriminator’s decision rule. Instead of estimating the absolute probability that individual samples are real, the discriminator in RaGAN predicts the likelihood that real samples are more realistic than the average of fake samples, and vice versa. This relative judgment enforces batchwise coupling between real and fake distributions in each update, yielding improved stability, convergence, and sample quality compared to standard GANs, while incurring negligible computational or architectural overhead (Jolicoeur-Martineau, 2018).

1. Mathematical Formulation and Theoretical Foundations

A standard GAN estimates $\mathbb{P}(x~\text{is real})$ for each individual $x$, using a discriminator $D(x)=\sigma(C(x))$, where $C(x)$ is an unconstrained critic and $\sigma$ is the sigmoid function. This “absolute” discrimination disregards the batch composition and ignores the prior that half the minibatch is typically fake.

RaGAN replaces this function with a relativistic probability. Let $\{x_{\mathrm{real}}\}$ from $p_{\mathrm{data}}$ and $\{x_{\mathrm{fake}}\}=G(z)$, where $G$ is the generator. The critic’s output for real and fake samples is compared with the average critic value of the opposing class:

• $$C_{\mathrm{RaGAN}}(x_\mathrm{real}, x_\mathrm{fake}) = \sigma(C(x_\mathrm{real}) - \mathbb{E}_{x_\mathrm{fake}}[C(x)])$$
• $$C_{\mathrm{RaGAN}}(x_\mathrm{fake}, x_\mathrm{real}) = \sigma(C(x_\mathrm{fake}) - \mathbb{E}_{x_\mathrm{real}}[C(x)])$$

The discriminator and generator losses are then: $$L_D = -\mathbb{E}_{x_\mathrm{real}}[\log C_{\mathrm{RaGAN}}(x_\mathrm{real}, x_\mathrm{fake})] - \mathbb{E}_{x_\mathrm{fake}}[\log(1-C_{\mathrm{RaGAN}}(x_\mathrm{fake}, x_\mathrm{real}))]$$

$$L_G = -\mathbb{E}_{x_\mathrm{real}}[\log(1-C_{\mathrm{RaGAN}}(x_\mathrm{real}, x_\mathrm{fake}))] - \mathbb{E}_{x_\mathrm{fake}}[\log C_{\mathrm{RaGAN}}(x_\mathrm{fake}, x_\mathrm{real})]$$

This formulation generalizes the relativism principle and can be extended to arbitrary $f$-divergences. For any concave $f$ with $f(0)=0$, $f'(0)\neq0$, and $\sup_x f(x)>0$, the objective

$$D^{\mathrm{Ra}_f}(P,Q) = \sup_{C:\mathcal{X}\to\mathbb{R}}\, \mathbb{E}_{x\sim P}\left[f(C(x) - \mathbb{E}_{y\sim Q}[C(y)])\right] + \mathbb{E}_{y\sim Q}\left[f(\mathbb{E}_{x\sim P}[C(x)] - C(y))\right]$$

is a bona fide statistical divergence (Jolicoeur-Martineau, 2019).

2. Motivation and Theoretical Properties

RaGAN addresses three principal shortcomings of the standard GAN:

• Batch-Level Relativity: Incorporates a priori knowledge that half the batch is fake by comparing each sample to the opposing class mean, restoring the correct equilibrium behavior for GAN discriminators.
• Divergence-Minimization and Gradients: Restores symmetric updates that push real and fake probabilities toward equilibrium, aligning non-IPM GANs with Integral Probability Metric GANs (e.g., WGAN) in terms of loss dynamics and discriminative gradients (Jolicoeur-Martineau, 2018).
• Coupling and Stability: By tying the outputs of all real and fake samples, RaGAN automatically dampens discriminator updates when one side lags, providing strong regularization against mode collapse and gradient saturation.

In standard GANs, D(x_real) can be driven to 1 independently of D(x_fake), leading to unsatisfactory discriminator gradients and persistent generator collapse. RaGAN’s batchwise coupling makes $D(x_\mathrm{real})$ decrease as $D(x_\mathrm{fake})$ increases, resolving this issue.

3. Practical Instantiations and Architectural Examples

Dual Relativistic Average GAN for Super-Resolution (POSR-GAN)

In single-image super-resolution, RaGAN can be implemented with dual discriminators—operating in both pixel and feature domains. Each branch uses the RaGAN loss with identical formulation but disjoint inputs:

• Pixel-Discriminator ($D^{\mathrm{pixel}}$): Inputs are upscaled RGB images (e.g., $128\times128$). Architecture: 8 Conv–BN–LeakyReLU blocks (channel doubling 64→512), two FC layers, single output (Ma et al., 2019).
• Feature-Discriminator ($D^{\mathrm{feature}}$): Inputs are VGG-19 conv4_4 features. Architecture: 7 Conv–BN–LeakyReLU blocks, spatial average pooling, single logit output.
• Generator: 128 Residual Channel Attention Blocks (RCAB) with shared-weight $3\times3$ convolutions, local/global skip connections, and sub-pixel convolution upsampling.

The combined generator loss, used after L1 pretraining, is

$$L_G^{\mathrm{total}} = L_{\mathrm{perceptual}} + \lambda L_1 + \eta_1 L_G^{\mathrm{pixel}} + \eta_2 L_G^{\mathrm{feature}}$$

with recommended hyperparameters $\lambda=10$, $\eta_1=\eta_2=0.125$.

Cross-Domain Wi-Fi Signal Denoising with Bi-LSTM RaGAN

For temporal signal adaptation (e.g., Wi-Fi amplitude spectra), both generator and discriminator can use Bi-LSTM backbones:

• Generator: LayerNorm → FC (52→256) → Bi-LSTM (256→512) → FC (512→52, per-step sigmoid). Output is a cleaned $1000\times52$ amplitude matrix.
• Discriminator: LayerNorm → Bi-LSTM (52→512) → FCs → sigmoid probability.

Training is performed on matched shielded/unshielded sequences, using the RaGAN loss in the temporal domain. The generator’s loss combines mean-squared and L1 content losses with the adversarial term (Avola et al., 29 Apr 2025).

4. Empirical Results and Performance Analysis

Empirical studies consistently demonstrate that RaGAN and its variants yield superior results compared to non-relativistic GANs:

Dataset	GAN Variant	FID (min, mean, SD)	Observations
CIFAR-10	SGAN	40.6	Baseline, unstable
CIFAR-10	RaSGAN	32.0	Large FID reduction
CIFAR-10	WGAN-GP	27.8	Requires 5 D-updates/G-update
CIFAR-10	RSGAN-GP	25.6	1:1 schedule, best FID
CAT 64x64	SGAN	Fails/Unstable	Mode collapse/oscillation in FID
CAT 64x64	RaLSGAN	min=11.97, mean=15.6, SD=2.6	Smoother convergence, lower variance
SR (PIRM-val)	SRGAN	PI ≈ 2.0	Baseline, high parameter count
SR (PIRM-val)	POSR-GAN	PI = 1.978	RaGAN, 30% SGAN params, sharper high-frequency detail

On cross-domain Wi-Fi signal denoising, RaGAN with bi-LSTM backbones yields a normalized MSE ≈ 0.19 between denoised and shielded spectra, and enables a downstream multi-class SVM to achieve 96% classification accuracy—significantly exceeding the baseline denoising autoencoder and standard CGAN (Avola et al., 29 Apr 2025).

5. Statistical Properties and Bias Considerations

When $f$ is non-linear, as in RaLSGAN ($f_{LS}(z)\!=\!-(z-1)^2+1$), the finite minibatch estimator of RaGAN is generally biased. The expected bias is $-\frac{1}{k}(\sigma^2_x+\sigma^2_y)$, where $\sigma^2_x$ and $\sigma^2_y$ are the critic’s variance on real and fake batches (Jolicoeur-Martineau, 2019). This bias shrinks as $O(1/k)$ as batch size increases, and explicit debiasing (using the unbiased estimator) does not improve, and may slightly degrade, perceptual quality as measured by FID.

Empirical results reveal that neither the minimum-variance unbiased estimator (RpGAN MVUE), nor bias removal for RaGAN, outperform the standard biased RaGAN estimator. The finite-sample bias and noise act as a regularizer that is favorable for gradient flow.

6. Optimization, Training Heuristics, and Implementation

• Update Ratio: RaGAN achieves state-of-the-art sample quality with a single D-update per G-update, contrasting with WGAN-GP that typically requires multiple D-updates per iteration (Jolicoeur-Martineau, 2018).
• Hyperparameters: Adam $(\beta_1=0.5, \beta_2=0.9\text{ or }0.999)$ with learning rates $1\text{e}^{-4}$–$2\text{e}^{-4}$. RaGAN performance is robust across various choices of activation functions and regularization schedules.
• Normalization: Use batch norm in the generator, spectral normalization in the discriminator for further gradient stabilization.
• Additional tricks: For very high-resolution synthesis or limited data, PacGAN and gradient penalty regularization can further suppress mode collapse and variance in sample quality.

The POSR-GAN demonstration (Ma et al., 2019) emphasizes parameter efficiency, with ∼5M parameters achieving parity or superior perceptual quality (PI) to architectures using more than triple the parameter count.

7. Broader Impact and Domain Applications

RaGAN has found impact in visual generative tasks (super-resolution, class-conditional synthesis), signal denoising, and cross-domain adaptation:

• In single-image super-resolution, dual-discriminator RaGAN enables matched or superior PI with substantially fewer parameters compared to ESRGAN/SRGAN and improves convergence stability (Ma et al., 2019).
• In cross-domain adaptation of Wi-Fi signals, RaGAN suppressed domain-dependent artifacts enabling robust downstream material classification—outperforming both autoencoders and conventional CGANs by 4–8 percentage points in accuracy (Avola et al., 29 Apr 2025).
• RaGAN’s stability and batchwise coupling are essential for reliable convergence, especially under small data or high-resolution settings where traditional GANs struggle or collapse (Jolicoeur-Martineau, 2018).

A plausible implication is that relativistic objectives may constitute a generalizable strategy for improving GAN performance in domains where classic per-sample discrimination leads to saturation or mode-collapse, especially for tasks involving perceptual detail or domain-adaptation under distribution shifts.

• "The relativistic discriminator: a key element missing from standard GAN" (Jolicoeur-Martineau, 2018)
• "On Relativistic $f$-Divergences" (Jolicoeur-Martineau, 2019)
• "Perception-oriented Single Image Super-Resolution via Dual Relativistic Average Generative Adversarial Networks" (Ma et al., 2019)
• "Digital Shielding for Cross-Domain Wi-Fi Signal Adaptation using Relativistic Average Generative Adversarial Network" (Avola et al., 29 Apr 2025)