Distance Correlation Disentanglement

• Distance correlation disentanglement is a deep learning method that uses dCorr to enforce true statistical independence between latent factors.
• It integrates differentiable dCorr/dCov loss terms into network training, enabling robust feature disentanglement and controlled attribute manipulation.
• Empirical evidence shows that this approach outperforms covariance-based methods in tasks like image synthesis, classification, and adversarial robustness.

Distance correlation disentanglement refers to a class of principled regularization methods in deep learning that leverage the statistical property of distance correlation (dCorr) to enforce independence between specified components of a learned representation. Unlike conventional covariance-based approaches, distance correlation possesses the zero-if-and-only-if-independence criterion, rendering it suitable for inducing true statistical independence between neural representations. This framework encompasses both distance covariance (dCov)-based penalties and their normalized forms (distance correlation), as well as extensions such as partial distance correlation (pdCorr) to control for nuisance variables. These regularizers have been employed in disentangled representation learning, fairness, robustness to adversarial attacks, and feature decorrelation, with empirical demonstrations across generative modeling and supervised classification tasks (Zhen et al., 2022, Song et al., 2020, &&&2&&&, Kasieczka et al., 2020).

1. Mathematical Foundations of Distance Correlation

Let $X\in\mathbb{R}^{p}$ and $Y\in\mathbb{R}^{q}$ be random vectors. The population distance covariance is defined via expectations over Euclidean norms:

$$\mathrm{dCov}^2(X,Y) = E[ \|X - X'\| \|Y - Y'\| ] + E[ \|X - X'\| ] E[ \|Y - Y'\| ] - 2 E\Big[ E[ \|X - X'\|\,|\,X ] E[ \|Y - Y'\|\,|\,Y ] \Big]$$

where $(X', Y')$ is an independent copy of $(X, Y)$. The corresponding sample version computes pairwise Euclidean distances for a mini-batch of $n$ paired samples $\{(x_i, y_i)\}_{i=1}^n$, with doubly-centered distance matrices $A_{ij}, B_{ij}$:

$$\widehat{\mathrm{dCov}^2(X,Y)} = \frac{1}{n^2}\sum_{i,j=1}^n A_{ij} B_{ij}$$

Distance correlation is then the normalized form:

$$\mathrm{dCor}(X,Y) = \frac{\mathrm{dCov}(X,Y)}{ \sqrt{\mathrm{dCov}(X,X) \mathrm{dCov}(Y,Y) } }$$

Crucially, $\mathrm{dCov}(X,Y) = 0$ if and only if $X$ and $Y$ are independent, irrespective of their linear or nonlinear relationship or dimensionality mismatch (Zhen et al., 2022, Kasieczka et al., 2020, Song et al., 2020, Song et al., 2019).

Partial distance correlation (pdCorr) extends this framework to cases where a nuisance variable $Z$ must be conditioned out. U-centered distance matrices are projected orthogonally to $Z$, yielding unbiased and orthogonalized estimates (Zhen et al., 2022).

2. Integration into Deep Neural Architectures

Distance correlation disentanglement is operationalized by incorporating dCorr or dCov-based penalties as differentiable loss terms in deep learning. The procedure is as follows:

• For latent codes $[f^1;\ldots;f^k]$ (factors of interest) and residual $r$, compute the mini-batch sample distance correlation $dCorr([f^1;\ldots;f^k], r)$.
• Add this as a regularization term to the full loss, e.g.,

$$L_{\text{total}} = L_{\text{rec}} + \lambda_{\text{cls}} L_{\text{cls}} + \lambda_{\text{ent}} L_{\text{ent}} + \lambda_{\text{res}} dCorr([f^1;\ldots;f^k], r)$$

where $L_{\text{rec}}$ is a reconstruction loss, $L_{\text{cls}}$ a classification loss, $L_{\text{ent}}$ an entropy term for unknown labels, and $L_{\text{res}}$ promotes factor/residual independence (Zhen et al., 2022).

• In the CDNet architecture, dCov is applied between the soft attribute code $\hat y$ and the style code $z$, while in mddAE it is inserted as $R_{\text{dcov}}(\hat Y, Z)$ (Song et al., 2020, Song et al., 2019).

The dCov/dCorr term is computed at each SGD step over the current mini-batch, with computational complexity $O(m^2 p)$ for batch size $m$ and feature dimension $p$. Backpropagation proceeds through all distance and centering operations (Zhen et al., 2022, Kasieczka et al., 2020). Approximations (random projections, block-wise estimation) exist but are not necessary for moderate batch sizes on modern accelerators.

3. Disentanglement via Statistical Independence

Disentangled representation learning aims for statistically independent semantic factors. Distance correlation disentanglement techniques exploit the zero-if-and-only-if-independence property of dCorr: driving $dCorr([f^1;\ldots;f^k], r)$ (or $dCov(\hat Y, Z)$) toward zero ensures that the designated attribute and residual subspaces are independent. This mechanism enforces that manipulation of attribute codes does not leak information through residual codes, enabling fine-grained, controlled, and physically coherent attribute editing (Zhen et al., 2022, Song et al., 2020, Song et al., 2019).

By contrast, conventional cross-covariance (XCov) penalties only enforce decorrelation in the second moment, leaving higher-order dependencies intact and failing to guarantee independence (Song et al., 2019, Song et al., 2020). Empirical results indicate that dCov-based regularization yields smoother, more isolated, and reliable attribute manipulations, and produces lower classification error on downstream disentanglement evaluations (Song et al., 2020, Song et al., 2019).

4. Representative Models and Training Protocols

Notable implementations of distance correlation disentanglement include:

• Partial dCorr regularization in StyleGAN2 for semi-supervised attribute disentanglement, as in Zhen et al. (2022), which demonstrates the disentanglement of age, gender, and hair color while freezing other features in the residual (Zhen et al., 2022).
• Controllable Disentanglement Network (CDNet), in which dCov enforces independence between attribute codes and complementary latent variables within a unified autoencoder/GAN hybrid architecture (Song et al., 2020).
• mddAE (multi-discriminator disentanglement autoencoder), employing an encoder-decoder framework with distance covariance regularization on soft attribute and style codes, using either sigmoid or softmax heads and explicit classification and reconstruction losses (Song et al., 2019).

In these approaches, dCov losses are typically ramped up over the early epochs to stabilize training, and mini-batch sizes of 64–128 yield stable optimization. At inference time, controllable image editing is possible by directly manipulating disentangled codes.

A typical classifier-based quantitative protocol is used to assess disentanglement: a downstream attribute classifier is trained, and the effect of controlled edits to $\hat y$ on classifier accuracy in reconstructed images is monitored. Lower classification error at controlled edits indicates stronger disentanglement (Song et al., 2019, Song et al., 2020).

5. Applications Beyond Disentanglement

Distance correlation regularization is not limited to disentangled representation learning. Additional applications include:

• Robustness and fairness: Penalizing dCorr between network output and nuisance variables (e.g., jet mass in LHC tagging, sensitive attributes in fairness-aware learning) yields predictors insensitive to unwanted correlations (Kasieczka et al., 2020, Zhen et al., 2022).
• Feature decorrelation across networks: Distance correlation can measure functional similarity between layers or between different network architectures, supporting systematic comparisons beyond layerwise metrics like canonical correlation analysis (Zhen et al., 2022).
• Adversarial robustness: Imposing independence between features or between different models increases resistance to adversarial perturbations by limiting information leakage across networks (Zhen et al., 2022).
• Partial dCorr for conditional decorrelation: Enables decorrelation while conditioning out complex confounders, supporting nuanced statistical control in representation learning pipelines (Zhen et al., 2022).

6. Computational Considerations and Practical Guidelines

Distance correlation regularization incurs an $O(m^2)$ cost per mini-batch due to pairwise distance computation. For batch sizes in the hundreds, the overhead is typically 15–25% per epoch and remains tractable on commodity GPU hardware (Zhen et al., 2022). Larger batch sizes improve estimation accuracy; $n \gtrsim 10^3$ is recommended for tasks with balancing decorrelation and supervised objectives (Kasieczka et al., 2020). The only free hyperparameter is the regularization strength $\lambda$, which smoothly interpolates between the baseline and fully decorrelated regimes.

The dCorr/dCov penalty is end-to-end differentiable and free from adversarial saddle point optimization, leading to stable training (Kasieczka et al., 2020, Song et al., 2020). It is plug-and-play with standard SGD or block stochastic gradient algorithms.

7. Empirical Results and Comparative Outcomes

Experiments on image synthesis benchmarks (FFHQ, CelebA) demonstrate that dCorr/dCov-based regularization achieves attribute-specific smooth editing, high reconstruction fidelity, and retention of object identity during joint attribute manipulations (Zhen et al., 2022, Song et al., 2020, Song et al., 2019). Compared to cross-covariance or adversarial decorrelation approaches, distance correlation regularization matches or outperforms in both decorrelation (disentanglement) and target task performance without introducing adversarial instability (Song et al., 2020, Kasieczka et al., 2020). In high energy physics tagging tasks, DisCo achieves state-of-the-art decorrelation quality relative to adversarial and heuristic baselines while offering algorithmic simplicity and stability (Kasieczka et al., 2020).

The zero-if-and-only-if-independence property and the differentiable, batch-wise computability of dCorr/dCov underpin its effectiveness and growing adoption in deep learning disentanglement and decorrelation tasks.