Transformed Residual Quantization (TRQ)

• TRQ is a framework that applies invertible transformations to residuals, aligning them for more effective quantization and reduced error.
• It uses techniques such as cluster-wise, subspace-preserving rotations and block-wise operations to adapt to data heterogeneity.
• TRQ has achieved notable gains in ANN search and neural network compression, lowering MSE and improving accuracy in low-bit regimes.

Transformed Residual Quantization (TRQ) is a general framework that enhances quantization by systematically transforming residuals prior to quantization, thereby improving approximation fidelity in both vector quantization and neural network compression scenarios. TRQ encompasses a range of instantiations, including local cluster-wise transformations in nearest neighbor search, structured subspace decompositions for quantization error reconstruction in deep networks, and rotation-based methods for robust residual handling under stringent bit constraints.

1. Conceptual Foundations and Problem Motivation

In classical quantization, the challenge is to efficiently represent high-dimensional data or neural network parameters with a small number of bits, minimizing distortion or accuracy loss. Traditional residual quantization (RQ) proceeds in stages: after an initial quantization, the residuals are further quantized. However, the distribution of these residuals can be ill-conditioned—randomly oriented or anisotropic—leading to suboptimal subsequent quantization. TRQ addresses this by applying invertible transformations to the residuals, aligning them into more structured forms to enable more effective downstream quantization.

The motivation for introducing TRQ is two-fold:

• Reducing Quantization Error: By transforming residuals, their distribution is made more homogeneous, lowering the tail energy and improving the efficacy of standard quantizers.
• Better Utilization of Bit/Ranks Budgets: In low-bit or low-rank regimes, preserving salient structures (such as high-energy singular components or important column directions) prior to quantization ensures that critical information is not lost in early quantization stages (Cho et al., 2 Feb 2026, Gu et al., 27 Jan 2026, Yuan et al., 2015, Li et al., 2017).

2. Methodological Frameworks of TRQ

TRQ frameworks are instantiated in several domains with distinct operational characteristics.

a. Local Transformations in Quantization Pipelines

In approximate nearest neighbor (ANN) search, two-level TRQ works as follows:

1. First Quantization Stage: An input vector $x_j\in\mathbb{R}^D$ is mapped to a codeword $c_i^1$ from the first-level codebook.
2. Residual Computation: The first-level residual $r_j = x_j - c_i^1$ is computed.
3. Cluster-Wise Transformation: For each first-level cluster $i$, an invertible linear transformation $T_i$ is learned. Residuals $V_i = \{ r_j: I_j^1 = i \}$ are aligned to a common shape $V_i' = \{ T_i r_j \}$.
4. Second-Level Quantization: Transformed residuals are quantized by a second-level quantizer $q_2(\cdot)$.
5. Reconstruction: Decoding requires applying the inverse transformation and summing the quantizations, $\hat{x}_j = c_i^1 + T_i^{-1}(q_2(T_i r_j))$.

The transformation $T_i$ is typically orthogonal (solved via the Procrustes problem), ensuring invertibility and norm preservation. Alternating minimization is employed to optimize both the second-level codebook and the set of transforms {T_i} (Yuan et al., 2015).

b. Structured Quantization for Deep Networks

In LLM quantization, TRQ is represented by Structured Residual Reconstruction (SRR):

1. Subspace Preservation: The top-$k$ singular subspace of a scaled weight matrix $\bSW = S W$ is isolated (via truncated SVD) and preserved exactly.
2. Residual Quantization: Only the residual $R = W - W_k$ orthogonal to the preserved subspace is quantized.
3. Low-Rank Error Reconstruction: The quantization error of the residual, $E_k = R - Q_k$, is further reconstructed via a low-rank ($r-k$) correction.
4. Rank Allocation: The split $k$ is chosen by minimizing a surrogate of the scaled reconstruction error; the unrecoverable energy in the residual versus the reconstructable energy is balanced using data-driven proxies.
5. Final Parameterization: $$\widehat W \approx Q + [L^{(1)}_k, L^{(2)}_k] [R^{(1)}_k; R^{(2)}_k]$$ with total rank budget $r$ (Cho et al., 2 Feb 2026).

c. Permuted Block-Wise Rotations

The LoPRo method utilizes block-wise permutation and partial rotations (Walsh-Hadamard) applied to residual matrices after low-rank decomposition. Salient column directions are preserved unrotated, while the remainder are grouped into rotation blocks, spreading out outliers and improving quantization performance in the sub-3-bit regime. Details include:

• Permutation based on the ratio of column-wise Hessian diagonals to mean residual magnitudes.
• Block-wise orthonormal rotations using the Hadamard basis.
• Group-wise scalar or vector quantization adapted per block.
• Mixed-precision storage of decomposition factors ($U$, $V$ in fp8, $\Sigma$ in fp16), yielding efficient storage and computation (Gu et al., 27 Jan 2026).

3. Mathematical Formulation and Optimization

Generalized TRQ Pipeline

TRQ operates on input matrices or high-dimensional vectors as follows:

1. Low-Rank/First-Level Quantization: $W \rightarrow W_r + R$
2. Transform Application: $R \rightarrow R T$ (where $T$ may be cluster-wise, block-structured, or data-adaptive)
3. Quantization: $R T \rightarrow Q(RT)$
4. Reconstruction: $W \approx W_r + Q(RT) T^{-1}$

Optimization objectives depend on the instantiation:

• For ANN search: mean-squared error (MSE) between original and reconstructed vectors.
• For neural networks: scaled or Hessian-weighted reconstruction error, often with subspace or block-importance constraints.

Selection of hyperparameters, such as the preserved subspace rank $k$, block sizes, or transformation types, is guided by problem-specific surrogate losses or theoretically motivated proxies (e.g., unrecoverable energy estimates, spectral density of random probes).

4. Complexity, Storage, and Computational Considerations

TRQ introduces moderate overhead compared to vanilla approaches, but achieves substantial quantization gains.

• Storage: Includes traditional codebooks plus storage for transforms (orthogonal or block-wise), which is acceptable for moderate cluster/block counts or when transforms are sparse/block-diagonal.
• Computational Overhead: The main additional cost is matrix-vector multiplications with $T_i$ or block transforms during assignment and reconstruction; these are typically negligible in ANN search settings since only a handful of clusters or blocks are visited per query (Yuan et al., 2015).
• Neural Network Inference: For LoPRo, overall cost scales as $\mathcal{O}(n b (2r+1+\log b_H))$ dominated by low-rank multiplications and fast Walsh–Hadamard transformations, with negligible memory and latency overhead in large $m,n$ regimes (Gu et al., 27 Jan 2026).

5. Empirical Performance and Applications

TRQ has demonstrated consistent empirical superiority to traditional quantization and product quantization (PQ) methods across domains:

Approximate Nearest Neighbor Search:

• On SIFT1M, GIST1M, and SIFT1B, TRQ reduces MSE by 10–30% and achieves absolute gains of 8–9% in Recall@1 over Optimized PQ with marginal additional compute and <3% overhead in storage (Yuan et al., 2015).
• This improvement is attributed to the ability of local residual transformations to reduce noise and align local distributions for more effective secondary quantization.

Neural Network Quantization and Compression:

• SRR achieves perplexity reductions up to 27.1% on LLaMA-2 7B and GLUE average gains of +5.9 points at 2 bits compared to QERA alone (Cho et al., 2 Feb 2026).
• LoPRo obtains near-fp16 accuracy on LLaMA-2/3 for 3-bit quantization and dramatically outperforms GPTQ in low-rank, low-bit regimes without fine-tuning (Gu et al., 27 Jan 2026).
• In binary-input networks, TRQ (with $m=2$ or $3$) achieves $\sim$20–30$\times$ speedup over floating-point baselines while matching or nearly matching their accuracy, outperforming simple thresholding (XNOR) by substantial margins (Li et al., 2017).

6. Limitations, Trade-offs, and Extensions

TRQ incurs additional storage due to the need for per-cluster or per-block transform matrices. For high $k_1$ or high-dimensional data, memory can become substantial, though restricting transforms to diagonal/block-diagonal or using fast representations (e.g., Hadamard or sparse rotations) mitigates this (Yuan et al., 2015, Gu et al., 27 Jan 2026).

Current approaches typically fix cluster codebooks or preserved subspaces and do not jointly optimize all components. Joint end-to-end optimization may further improve fidelity but increases algorithmic complexity. For applications involving more than two quantization stages, the compound effect of deep chains of transforms raises open questions on compute/distorion trade-offs.

Extensions of TRQ to broader quantization targets (activation quantization, key/value cache quantization) and kernel fusion at inference time have been suggested (Gu et al., 27 Jan 2026).

A plausible implication is that, as neural and retrieval models continue to scale and the pressure for high-fidelity compression intensifies, TRQ-style frameworks—emphasizing transformation of heterogeneous residual subspaces—will be increasingly central in low-bit and high-throughput regimes.

7. Historical Evolution and Context Across Research Domains

TRQ was introduced in vector quantization and ANN search to overcome limitations of RQ and PQ in clustering heterogeneous or anisotropic data (Yuan et al., 2015). The principle was later adapted to neural networks, first in high-order binary network acceleration (Li et al., 2017), then in mixed-precision, block-rotation schemes for neural weight or activation quantization (Gu et al., 27 Jan 2026), and most recently in subspace-preserving, theory-guided rank-allocation frameworks for LLM quantization and parameter-efficient fine-tuning (Cho et al., 2 Feb 2026).

Throughout these methodological progressions, TRQ has demonstrated the generality and efficacy of leveraging data-dependent or structure-aware transformations of the residual signal as a critical step prior to quantization, enabling both optimal error decay and preservation of functional capacity at minimal cost.