REVQ: Adaptive Residual Experts Quantization

• REVQ is an adaptive quantization strategy that decouples codec capacity from bitrate by combining a shared base codebook with dynamically selected expert codebooks.
• It uses a learned routing mechanism to allocate sparse quantization resources based on input complexity, ensuring efficient compression and high-fidelity audio reconstruction.
• Empirical results show that REVQ outperforms traditional fixed-depth RVQ, achieving superior latent reconstruction accuracy and scalable bitrate-quality performance.

Residual Experts Vector Quantization (REVQ) is a quantization strategy designed for neural audio coding under tight bitrate constraints. It combines a shared base codebook with a large pool of dynamically routed expert codebooks, enabling adaptive sparse quantization of encoded audio latents. REVQ decouples codec representational capacity from per-segment bitrate, substantially improving both compression efficiency and fidelity, especially in low-bandwidth settings. It forms the core of high-fidelity neural audio codecs such as SwitchCodec, surpassing fixed-depth residual vector quantization (RVQ) approaches by adaptively allocating quantization resources as required by the input complexity (Wang et al., 28 Jan 2026, Wang et al., 30 May 2025).

1. Motivating Principles and Conceptual Distinctions

Standard residual vector quantization (RVQ) employs a fixed cascade of $M$ codebooks on every frame, which is inefficient for content with variable complexity—simple segments are over-encoded while complex signals become under-represented at restricted bitrates. REVQ addresses this by introducing:

• A shared "base" quantizer capturing the predominant structure of each latent vector.
• A pool of $N_n$ expert quantizers, of which only $k_n \ll N_n$ are selected per audio segment via a learned routing mechanism.
• Sequenced application of the shared quantizer and the selected experts, sorted in ascending index to respect the residual energy hierarchy.

This architecture decouples the codebook capacity from the bitrate; $N_n$ governs the potential granularity, while $k_n$ directly controls bitrate, making quantization allocation adaptive, efficient, and highly granular.

2. Formal Architecture and Encoding Pipeline

REVQ leverages two quantizer families:

• Shared Quantizer $Q_0$: Codebook $E_0 \in \mathbb{R}^{K_0 \times D}$, addressing the "coarse" structure of the latent $Z$.
• Expert Quantizers $Q_{1\dots N_n}$: Codebooks $E_i \in \mathbb{R}^{K_i \times D}$ for $i=1\dots N_n$, specializing in progressive residual refinement.

The encoding process for each window of latent frames $Z \in \mathbb{R}^{T\times D}$ proceeds as follows:

1. Affinity computation: Average the latents over the window and project using a learnable bias-free matrix $U \in \mathbb{R}^{N_n \times D}$ to obtain expert scores:

$$S = \frac{1}{T} \sum_{t=1}^T Z_t U^\top \in \mathbb{R}^{N_n}$$

1. Expert selection: Select the top-$k_n$ entries of $S$ to form a binary mask $\text{mask}_i \in \{0,1\}^{N_n}$ indicating chosen experts.
2. Quantization: Apply $Q_0$ on the original latent, then sequentially (in sorted index order) apply chosen experts on the residuals.

The mask and quantization indices comprise the bitstream. The mask cost is approximately $\lceil \log_2 \binom{N_n}{k_n} \rceil$ bits per window (Wang et al., 28 Jan 2026).

3. Mathematical Specification

The residual quantization for each frame $t$ is formally:

• Let $Z^0 = Z_t$.
• Shared stage:

$$k_0(t) = \arg\min_j \|Z^0 - e_{0,j}\|_2, \quad Z_{0q,t} = e_{0,k_0(t)}, \quad R^1_t = Z^0 - Z_{0q,t}$$

• Expert stages: Let $I = \{i\ |\ \text{mask}_i = 1\}$, sorted ascending. For $m=1 \dots k_n$ (with $i_m$):

$$k_{i_m}(t) = \arg\min_j \|R^m_t - e_{i_m,j}\|_2,\quad Z_{i_m,q,t} = e_{i_m, k_{i_m}(t)},\quad R^{m+1}_t = R^m_t - Z_{i_m, q, t}$$

• Final output:

$$Z_{q,t} = Z_{0q,t} + \sum_{m=1}^{k_n} Z_{i_m, q, t}$$

Gradient flow through the selection mask is enabled by a Straight-Through Estimator (STE) (Wang et al., 28 Jan 2026).

4. Training Objectives and Regularization

The end-to-end objective includes a waveform-level reconstruction loss, typically involving $\ell_1$ distance and multi-scale STFT loss, potentially supplemented with adversarial objectives via multi-tiered STFT and waveform discriminators ("MTSD", "MPD") (Wang et al., 30 May 2025). The only additional regularization is the optional expert-balance penalty (cross-entropy or Gini), though SwitchCodec omits this.

No explicit bitrate penalty is imposed on expert usage; the router learns an optimal sparse selection in service of reconstruction fidelity. For expert load balancing and codebook utilization (preventing routing collapse), a bias update mechanism—Developing Router Protection Strategy (DRPS)—boosts affinities of underused experts, maintaining competitive but balanced expert selection (Wang et al., 30 May 2025). This is implemented with a bias $b_i$ for affinity scores, increased if expert $i$ is underused, reset if overused, and otherwise maintained, with $\gamma \sim 0.01$ as step size.

5. Variable-Bitrate Control and Inference Dynamics

REVQ's gating mechanism allows variable-bitrate operation via simple adjustment of $k_n$ at inference; lower $k_n$ yields coarser but lower-rate quantization, higher $k_n$ produces finer high-fidelity reconstructions. No retraining is required; bitrate is controlled entirely via router configuration.

A single SwitchCodec model demonstrates effective coverage of 0.89–8 kbps bandwidth simply by varying $k_n$; subjective (MUSHRA) and objective (PESQ, Mel distance, ViSQOL, STFT distance) metrics improve monotonically with $k_n$ (Wang et al., 28 Jan 2026). This suggests that expert selection affinity correlates with signal complexity and enables smooth bitrate-quality scaling.

6. Empirical Findings and Ablation Studies

Key benchmarks demonstrate the superiority of adaptive REVQ compared to fixed-depth RVQ and other sparse quantization baselines:

• At fixed $k_n=3$ experts, adaptive REVQ achieves 17.6% higher latent-reconstruction accuracy than using the first 3 experts in fixed index order (Wang et al., 28 Jan 2026).
• As $N_n$ increases (5 to 17), the fraction of activated experts drops (100% to 16.6%), but objective metrics (PESQ, Mel loss, ViSQOL) remain nearly constant, confirming efficient expert selection.
• At ≈2.7 kbps, SwitchCodec (REVQ + MTSD) achieves PESQ 2.87, Mel distance 0.75, ViSQOL 4.27, outperforming DAC and EnCodec at comparable rates. Ablation studies further show quantitative improvements when routed experts and the MTSD discriminator are employed (Wang et al., 30 May 2025).

Configuration	PESQ	Mel Dist	ViSQOL	Expert Usage (%)
5 experts, no DRPS	2.53	0.83	3.92	100.0
9 experts, no DRPS	2.57	0.82	3.94	44.4
17 experts, no DRPS	2.57	0.81	3.92	16.6

A plausible implication is that REVQ’s router efficiently concentrates quantization on structurally significant segments, maintaining quality even as available expert pool increases.

7. Implementation, Bandwidth, and Operational Trade-offs

The bitstream includes indices for all quantized latents (shared and experts) as well as the expert-selection mask. For non-streaming setups (long segments), the mask overhead ($N_n$ bits per window) is negligible (e.g., 9 bits over 2 s $\to$ 4.5 bps). In real-time operation (10 ms frame), mask costs ($100 N_n$ bps for $N_n$ experts) may require slightly higher base rates (2.3 kbps vs. 1.5 kbps).

Hyperparameters for DRPS, particularly the bias increment $\gamma$, must remain small to prevent collapse or excessive codebook homogenization. Codebooks should be sorted by granularity, with lower-index experts handling high-energy residuals to ensure stable progressive quantization.

Summary

Residual Experts Vector Quantization (REVQ) provides a dynamic, adaptive alternative to fixed-depth RVQ, pairing a base quantizer with a sparse, router-selected subset of expert quantizers. This approach yields bit-efficient, adaptive fidelity across diverse audio content, empowers seamless bitrate scaling at inference, and achieves state-of-the-art metrics at extreme compression levels. REVQ forms the quantization backbone of SwitchCodec and similar codecs, using expert routing, DRPS strategies, and MTSD adversarial training for robust spectral fidelity (Wang et al., 28 Jan 2026, Wang et al., 30 May 2025).