Dual-Path VQ-VAE: Disentangled Generation

• The paper introduces dual-path VQ-VAE, which separates geometry and colour attributes to enable fine-grained, controllable output generation.
• It employs independent quantisation and an auxiliary skip regularisation to ensure each branch captures complementary, non-redundant features.
• Empirical results demonstrate lower FID and improved precision/recall compared to VQ-GAN, confirming enhanced structural consistency and fidelity.

Dual-Path VQ-VAE models are structured extensions of vector quantised variational autoencoders designed to disentangle and separately encode distinct generative factors within data. The central paradigm is to route semantically orthogonal attributes—such as image colour and geometry, or speech suprasegmental prosody and phone content—along parallel encoding branches with independent quantisation bottlenecks, supporting controllable generation, attribute transfer, and robust sampling. These models instantiate architectural and objective constraints to ensure that each branch captures complementary, non-redundant information, thereby facilitating fine-grained control over composite outputs and improving interpretability as well as performance metrics on diverse datasets (Rathakumar et al., 2023).

1. Architectural Principles: Dual Branches and Bottlenecks

Dual-path VQ-VAE is characterized by a bifurcated latent representation pipeline, generally composed of two processing branches:

• Geometry branch: Processes the structural or intensity-related attributes of the input. For images, an “intensity module” (deep 3×3 convolutions) maps the input $x \in \mathbb{R}^{H \times W \times 3}$ to a grayscale-like map $g \in \mathbb{R}^{H \times W \times 1}$.
  • A VQ-encoder $E_g$ then compresses $g$ into lower resolution $f_g \in \mathbb{R}^{h \times w \times d}$, which is quantised through its dedicated codebook $Q_g$ to produce indices $z_g$.
• Colour branch: Encodes the residual (non-structural) attributes, such as colour information.
  • Typically employs a continuous encoder $E_c$ that produces a latent Gaussian $q(z_c|x) = \mathcal{N}(\mu_c, \sigma_c^2)$ in place of quantisation.

The decoder operates hierarchically, with skip connections from the quantised geometry code $z_g$ and the colour latent $z_c$ injected into each upsampling stage via “merge modules.” Each merge module $M_k$ concatenates projections from both branches, followed by convolutional refinement, ensuring that structurally salient details are sourced from $z_g$ while colour features are modulated by $z_c$ (Rathakumar et al., 2023).

2. Disentanglement Objectives and Regularisation

Disentanglement is enforced through both architectural wiring and training objectives:

• Skip-connection bias: Geometry/structure tokens $z_g$ are provided as direct skip connections to early decoder layers. This biases the network to attribute all spatially precise, high-frequency variation (e.g., edges, contours) to $z_g$ and to reserve $z_c$ for global, residual statistics (e.g., colour).
• Auxiliary skip regularisation: Dual-path VQ-VAE introduces a novel auxiliary reconstruction loss. During training, the decoder is asked to reconstruct the input not only using the true skip-connected latents, but also using the encoder-side features in their place. This loss penalizes solutions in which one latent is ignored and ensures both branches remain active, addressing the common collapse of one-latent multi-head VAE architectures.
• Objective function: The full loss comprises

$$L = L_{\text{recon}} + L_{\text{aux}} + D_{\text{KL}}(q(z_g|g) \;\|\; p(z_g)) + D_{\text{KL}}(q(z_c|x) \;\|\; p(z_c)) + \lambda_g L_{\text{commit}}(z_g) + \lambda_c L_{\text{commit}}(z_c)$$

where $L_{\text{recon}}$ and $L_{\text{aux}}$ include $\ell_1$ and LPIPS losses, and commitment losses are as in standard VQ-VAE.

This combined mechanism produces an implicit ELBO that, empirically, yields strong disentanglement and faithful, attribute-controlled generation (Rathakumar et al., 2023).

3. Attribute Control and Conditional Generation

Dual-path architectures facilitate direct manipulation and transfer of specific generative attributes.

• Controlled palette transfer: To generate samples with a user-specified palette, an exemplar image is encoded to extract its colour latent $z_c^{ex}$. New geometry tokens $z_g$ are sampled from the learned AR prior. The decoder then synthesizes images that pair the exemplar’s colour statistics with novel or fixed structure.
• Recolouring (REDUALVAE variant): For attribute transfer between real images, the geometry branch encodes a target image’s structure, while the colour latent is sourced from another image. This preserves the former’s spatial arrangement while transplanting the latter’s colour.
• Unconditional generation: Sampling both latents from their priors yields plausible but uncontrolled outputs.

Qualitative results indicate that conditioned samples preserve fine structure and allow gradual, semantically coherent shifts in colour space through linear interpolation of the colour latent (Rathakumar et al., 2023).

4. Training Regimes and Evaluation Metrics

Training employs conventional stochastic optimization and self-supervised objectives alongside the dual-path enhancements.

• Datasets: Experiments span a wide range of distributional complexities, including AnimeFaces, MetFaces, artistic landscapes, logos, CUB Birds, flowers, and animal faces, with typical 95/5 train/test splits.
• Optimisation: Adam with learning rate $5 \times 10^{-4}$, $\beta_1=0.9$, $\beta_2=0.999$, and $\epsilon=10^{-8}$; batch size 64.
• Architectural hyperparameters: Codebook size $N_g=512$, code dimension $d_g=256$, colour latent dimension $d_c=128$, AR prior as a 16-layer Transformer.
• Reconstruction and regularisation: Training loss combines $\ell_1$ pixel error, LPIPS, commitment losses, and the auxiliary skip-path loss.
• Evaluation: Sample quality is measured using Fréchet Inception Distance (FID), Precision/Recall on samples, and a custom 2D colour-histogram KL divergence for colour transfer fidelity.

The dual-path regulariser produces lower FID and better colour fidelity scores compared to single-branch VQ-VAE and VQ-GAN baselines, and ablations confirm that the auxiliary loss is essential to achieving disentangled and controllable outputs (Rathakumar et al., 2023).

Dataset	VQ-GAN FID	DualVAE FID	DualVAE Precision/Recall
Birds	13.8	13.8	0.959 / 0.946
Logos	49.6	25.5	0.973 / 0.962
Art Lands.	30.4	18.0	0.951 / 0.975
Butterflies	23.9	19.5	0.979 / 0.981
MetFaces	65.5	27.0	0.897 / 0.907

Lower FID and higher Precision/Recall with DualVAE indicate improved generative performance.

5. Empirical Results and Qualitative Phenomena

Dual-path VQ-VAE demonstrates:

• Palette transfer fidelity: Conditional image generation anchored to exemplar palettes yields low KL divergence between reference and generated colour histograms; auxiliary skip regularisation reduces KL from ~0.94 (without) to ~0.68 (with), well below random test–test baseline of 0.98.
• Structural consistency: Geometry codebooks and skip pathways preserve object boundaries and salient edges across diverse content domains.
• Smooth attribute interpolation: Linear traversals in $z_c$ space morph colour schemes smoothly while maintaining fixed geometry.
• Semantic recolouring: Recolouring of logos, illustrations, and landscapes is semantically plausible; for example, sky or foliage regions retain their context when their colours are replaced.

A plausible implication is that such models enhance fine-grained artistic and design workflows, especially in domains where explicit factor control and attribute manipulation are paramount (Rathakumar et al., 2023).

6. Limitations and Future Prospects

Documented limitations include:

• Colour-fidelity bottleneck: The palette matching is limited by histogram-based measurement and the complexity of the colour codebook. Highly intricate or multi-object images may present challenges.
• Geometry diversity cap: The geometry codebook captures only the vocabulary seen during training; the model cannot sample out-of-distribution geometry unless extended with more flexible priors (e.g., hierarchical or stochastic codebooks).
• No explicit text or abstract attribute control: The current architecture does not support conditioning on non-visual attributes such as textual description or higher-level semantics.
• Prospective enhancements: Integration with diffusion priors or cross-modal encoders is suggested to enable richer generative controls and out-of-distribution attribute transfer.

The separation of geometry and colour latents also provides a foundation for future extensions involving multi-modal, variable-length, or more densely factorized representations (Rathakumar et al., 2023).