Variational Deterministic Information Bottleneck

• VDIB is an information-theoretic framework that replaces mutual information regularization with a direct entropy penalty to learn discrete, compressed representations.
• It underpins the VQ-VAE loss formulation by aligning deterministic quantization with a zero-entropy code and incorporating squared-error terms for codebook and commitment alignment.
• The method balances reconstruction fidelity and compression strength, providing a precise trade-off control between latent code diversity and signal preservation.

The Variational Deterministic Information Bottleneck (VDIB) is an information-theoretic principle for learning discrete, compressed representations in autoencoders, specifically providing the theoretical foundation for Vector Quantized-Variational Autoencoders (VQ-VAE). VDIB originates by variationally approximating the Deterministic Information Bottleneck (DIB) objective, substituting the explicit mutual information regularization of classical Information Bottleneck (IB) with a direct penalty on the entropy of the representation. In the discrete latent setting of VQ-VAE, the VDIB framework leads to a precise loss formulation: the sum of expected negative log-likelihood (reconstruction error) and a cross-entropy term penalizing codebook usage according to a reference prior. Deterministic quantization yields a zero-entropy code, producing a loss structure matching the canonical VQ-VAE objective.

1. Information Bottleneck Frameworks: IB and DIB

The original Information Bottleneck (IB) method seeks a stochastic encoder $p(z\mid i)$ that compresses $I$ (data indices) to a code $Z$, minimizing mutual information $I(I;Z)$ subject to maintaining predictive relevance for the variable $X$. The objective is: $$L_{\rm IB}[p(z \mid i)] = d_{\rm IB}(I,Z) + \beta I(I;Z)$$ where

$$d_{\rm IB}(I, Z) = \mathrm{KL}[p(X \mid I) \| p(X \mid Z)]$$

measures distortion—how much predictive information about $X$ is lost through $Z$.

The Deterministic Information Bottleneck (DIB) modifies this by replacing $I(I;Z)$ with the Shannon entropy of $Z$, $H(Z)$: $$L_{\rm DIB}[p(z \mid i)] = d_{\rm IB}(I, Z) + \beta H(Z)$$ where $H(Z) = -\sum_{z} p(z) \log p(z)$ and $p(z)$ is the marginal over encodings. The tightening $H(Z) \leq I(I;Z)$ allows DIB to yield simpler, more clustered representations.

2. The VDIB Objective: Variational Bounds and Formulation

In practical autoencoders, $p(z \mid i)$ is parameterized by neural encoders and both $p(x \mid z)$ and $p(z)$ are intractable. VDIB introduces variational approximations:

• $q(x \mid z)$ as the decoder approximating $p(x \mid z)$,
• $r(z)$ as a prior approximating the marginal $p(z)$.

The variational bounds are:

• For distortion:

$$d_{\rm IB}(I, Z) \leq \mathbb{E}_{p(i,z,x)}[-\log q_\phi(x \mid z)]$$

• For entropy:

$H(Z) \leq H(p(z), r(z)) = -\sum_z p(z) \log r(z)$

Thus, the VDIB objective is: $$L_{\rm VDIB} = \mathbb{E}_{p(i,x)}\mathbb{E}_{p(z \mid i)}[-\log q_\phi(x \mid z)] + \beta H(p(z \mid i), r(z))$$ In expanded form for discrete $K$-way latent codes and $N$ datapoints: $$L_{\rm VDIB} = -\sum_{i=1}^N \sum_{z=1}^K p(z \mid i)\log q_\phi(x_i \mid z) + \beta \sum_{i=1}^N \sum_{z=1}^K p(z \mid i)[-\log r(z)]$$ With uniform $r(z)$, the cross-entropy term reduces to $\beta \sum_i H(p(z \mid i))$.

3. Derivation of the VQ-VAE Loss from VDIB

VQ-VAE instantiates VDIB with deterministic quantization:

• Encoder $z_e(x_i)$ maps input $x_i$ to $\mathbb{R}^D$.
• Codebook $\{e_j\}_{j=1}^K$ stores discrete embeddings.
• Nearest-neighbor quantization: $z_i = \arg\min_j \|z_e(x_i) - e_j\|^2$; $p(z \mid i) = \delta(z, z_i)$.

This yields $H(p(z \mid i)) = 0$, causing the entropy term to vanish (aside from a constant). The reconstruction term remains: $$\frac{1}{N}\sum_{i=1}^N [-\log q_\phi(x_i \mid z_q(x_i))]$$

VQ-VAE adds two squared-error terms using the stop-gradient operator $\mathrm{sg}(\cdot)$ to overcome quantizer non-differentiability: $$L_{\rm VQ-VAE} = \underbrace{\sum_i -\log q_\phi(x_i \mid z_q(x_i))}_{\rm recon} + \underbrace{\beta \|\mathrm{sg}[z_e(x_i)] - z_q(x_i)\|^2}_{\rm codebook} + \underbrace{\beta \|z_e(x_i) - \mathrm{sg}[z_q(x_i)]\|^2}_{\rm commitment}$$

These augmentations enable robust encoder-codebook alignment and code assignment commitment, operationalizing the VDIB principle in deep neural architectures.

4. Roles of Encoder, Quantizer, and Decoder

• Encoder ($f_{\theta_e}$): Maps input $x_i$ to a $D$-dimensional latent vector $z_e(x_i)$. Encodes as a deterministic delta distribution (nearest neighbor) in standard VQ-VAE; or a “soft” distribution $p_\theta(z \mid i)$ in EM-style extensions.
• Quantizer/Bottleneck: Enforces a discrete codebook partitioning. Deterministic (VDIB) quantization assigns each $z_e(x_i)$ to one $e_j$. Soft quantization (VIB) assigns codewords by similarity-based probabilities, raising code entropy.
• Decoder ($q_{\theta_o}(x \mid z)$): Implements $q(x \mid z)$, maximizing overall reconstruction likelihood.

5. Compression–Reconstruction Trade-off and Lagrange Multiplier $\beta$

VDIB loss comprises two conceptually distinct terms:

• Distortion (Reconstruction Error): $\mathbb{E}_{p(i,z,x)}[-\log q(x \mid z)]$ quantifies loss of information about $x$ after compression via $z$.
• Compression (Entropy Penalty): $\beta H(p(z \mid i), r(z))$ regulates the compactness of $Z$.

The parameter $\beta$ modulates this trade-off:

• $\beta \to 0$ collapses to plain (unregularized) autoencoding—maximal code diversity, minimal compression.
• $\beta \to \infty$ enforces extreme compression—minimal code entropy, risking information loss.

The rate-distortion balance is identical to rate–distortion theory.

6. Empirical and Theoretical Observations

• Zero-Entropy Encoding: In original VQ-VAE, nearest-neighbor assignment makes $H(p(z \mid i))=0$. Compression is purely determined by codebook cardinality $K$.
• Soft Quantization and Perplexity: EM-based VQ-VAE generalization replaces $\delta$-assignment with soft probabilities,

$$p(z \mid i) = \frac{\exp(-\|z_e(x_i)-e_z\|^2)}{\sum_{j} \exp(-\|z_e(x_i)-e_j\|^2)}$$

increasing $H(p(z \mid i))$. This is a VIB, not VDIB, model, showing higher latent perplexity and richer codeword utilization at some cost in rate.

• Theoretical Framework: The main theoretical result in Wu & Flierl (2018) (Wu et al., 2018) is that VQ-VAE with deterministic quantization and uniform prior precisely implements VDIB, while EM-based extensions correspond to VIB with strictly positive entropy.

7. Connection and Significance

VDIB formalizes the objective underlying discrete-latent autoencoders:

• One minimizes expected reconstruction error plus a cross-entropy regularizer on code assignments.
• In the deterministic nearest-neighbor case with a uniform prior, this matches the VQ-VAE loss exactly.
• The approach provides a clear information-theoretic interpretation for composition and training of discrete latent models, enabling precise control over trade-offs between compactness and predictive fidelity.

The VDIB framework offers theoretical clarity and practical guidance for designing discrete bottlenecks in neural architectures, explaining observed empirical phenomena such as codebook utilization and latent perplexity, and illuminating the distinctions between deterministic and stochastic approaches to representation learning (Wu et al., 2018).