Geometry-Induced Query-Key Transformation (GIQT)

• GIQT is a family of methods that integrates geometric metadata into transformer attention to correct anisotropic distortions in query and key spaces.
• It employs techniques like polar coordinate transformation and hyperspherical normalization to improve quantization and cross-view vision performance.
• Empirical results show that GIQT boosts accuracy and efficiency in applications such as cross-view re-identification and language modeling with minimal overhead.

Geometry-Induced Query-Key Transformation (GIQT) refers to a family of methods that modify or augment the computation of attention similarity in neural architectures—primarily transformers—by incorporating or exploiting the underlying geometric structure of the query and key spaces. This approach spans both explicit geometry-aware similarity rectification, as in aerial-ground cross-view vision, and geometric reparameterizations for computational efficiency, as in quantized attention in LLMs. Representative instances include the GIQT rectification module for cross-view person re-identification (Hambarde et al., 29 Jan 2026), the polar-coordinate (geometry-induced) transformation in PolarQuant (Wu et al., 1 Feb 2025), and the use of hyperspherical normalization in QKNorm (Henry et al., 2020).

1. Mathematical Foundations and Motivations

The underlying premise of GIQT is that the standard attention dot-product $q \cdot k$ assumes a Euclidean, geometry-invariant similarity measure, which is insufficient under transformations that induce distributional shifts or distortions in embedded features. In many applications, such as long-context quantization (Wu et al., 1 Feb 2025) or cross-view matching where camera geometry induces strong anisotropy (Hambarde et al., 29 Jan 2026), this assumption breaks down.

Specifically:

• In the context of rotary position embeddings (RoPE), input feature vectors are partitioned into 2D subspaces and rotated, yielding a natural polar coordinate structure for each pair $(x_j, y_j)$. Quantities of interest become the radius $r_j = \sqrt{x_j^2 + y_j^2}$ and angle $\theta_j = \mathrm{atan2}(y_j, x_j)$ (Wu et al., 1 Feb 2025).
• For cross-view vision, attention similarities between queries (Q) and keys (K) are systematically distorted by view geometry, leading to anisotropic similarity spaces that no longer align across camera viewpoints. GIQT introduces an adaptive linear transform $T(e_{\mathrm{geo}})$, parameterized by geometric metadata, to rectify these distortions prior to dot-product computation (Hambarde et al., 29 Jan 2026).
• In normalization-based approaches (e.g., QKNorm), projecting $Q$ and $K$ onto the unit hypersphere induces a cosine similarity geometry on queries/keys, providing boundedness and improved stability for softmax attention (Henry et al., 2020).

2. Core Methodologies

2.1 Geometry-Conditioned Similarity Rectification

In cross-view re-identification, GIQT operates by:

• Collecting geometric side-information: altitude ($h$), lateral distance ($d$), tilt angle ($\theta$), and discrete camera ID ($c$).
• Embedding and concatenating these into a geometry feature vector $e_{\mathrm{geo}} \in \mathbb{R}^{4 d_g}$.
• For each transformer layer, applying a low-rank, geometry-conditioned linear transform $$T(e_{\mathrm{geo}}) = I_d + U(e_{\mathrm{geo}}) V(e_{\mathrm{geo}})^\top$$ where $$U,V: \mathbb{R}^{4 d_g} \rightarrow \mathbb{R}^{d \times r}$$ (for a small $r$).
• Transforming queries and keys: $Q' = Q T(e_{\mathrm{geo}})$, $K' = K T(e_{\mathrm{geo}})$.
• Computing rectified attention: $s_{ij}' = {Q'_i}^\top K'_j = Q_i^\top T^\top T K_j$.

The above introduces correction terms in the directions of principal geometric distortion, enabling anisotropy compensation without modifying the base feature extractor or attention formulation (Hambarde et al., 29 Jan 2026).

2.2 Polar Coordinate Transformation for Quantization

In quantized caches for sequence models, GIQT refers to:

• Decomposing each $d$-dimensional vector into $d/2$ two-dimensional sub-vectors.
• Representing each sub-vector $(x_j, y_j)$ in polar coordinates $(r_j, \theta_j)$.
• Quantizing $r_j$ using a uniform $n$-bit quantizer over $[0, R_j^{\max}]$ and $\theta_j$ into $2^m$ bins over $(-\pi, \pi]$.
• Storing for each channel lookup tables for $\cos_j[a], \sin_j[a]$ for $a \in \{0, \ldots, 2^m-1\}$ to reconstruct $k_{2j}, k_{2j+1}$ efficiently.
• At decoding, computing inner products via table lookup and small vector operations rather than by restoring full-precision vectors (Wu et al., 1 Feb 2025).

This geometry-induced parameterization allows effective quantization even with extreme outliers, as outlier values typically affect only one dimension in the 2D block but not the corresponding polar radius.

2.3 Geometry-Induced Normalization

QKNorm provides another instance of geometry-induced query-key transformation:

• $\ell_2$-normalizing $Q$ and $K$ vectors, mapping them to the unit hypersphere, thus restricting attention similarity to cosine angles.
• Introducing a global learnable scaling parameter $\alpha$ (or $g$), replacing fixed division by $\sqrt{d}$ with data-driven adaptation.
• Computing attention as $A = \mathrm{softmax}(g\, \hat Q \hat K^\top)$ (Henry et al., 2020).

This method drives the geometry of attention computation explicitly towards angular similarity, decoupling representation magnitude from similarity space.

3. Algorithmic Implementation Details

Implementation protocols for GIQT variants depend on context but share several properties:

• Low-rank update (vision GIQT): Uses small MLPs per block ($\mathrm{MLP}_U$, $\mathrm{MLP}_V$) to produce $U$, $V$ factors, enabling efficient $Q' = Q + (Q V)U^\top$ and $K' = K + (K V)U^\top$ without materializing large $d \times d$ matrices.
• PolarQuant acceleration: Key decoding combines per-block table lookups (for angle) and recovering $k_{2j} = r_j \cos \theta_j$, $k_{2j+1} = r_j \sin \theta_j$ with only two multiplications and accumulations per block.
• Geometry-conditioned prompts (Hambarde et al., 29 Jan 2026): Parallel prompt tokens are constructed as $$P_{\mathrm{geo}} = P_{\mathrm{base}} + \alpha \,\Delta P$$, where $\Delta P$ depends on global view-invariant features and geometry embedding.
• Auxiliary losses and regularization: Cross-entropy and triplet losses are applied to both global and geometry-rectified features, with additional classification and orthogonality regularization terms to encourage disentanglement of view-specific and invariant factors.

4. Performance Characteristics and Empirical Results

Empirical evaluations confirm that GIQT modules deliver both accuracy and computational benefits:

• For aerial-ground re-identification, addition of GIQT yields gains of 0.64–3.75% absolute Rank-1 and 1.16–4.42% mAP across multiple benchmarks (e.g., AG-ReID, CARGO, DETReIDX), with under 3% extra FLOPs per decoder layer and negligible parameter count compared to standard ViT-B (Hambarde et al., 29 Jan 2026).
• In PolarQuant, geometry-induced polar quantization reduces key-cache memory usage to 4.16 bits per dimension (vs. 5.08 for KIVI and 4.32 for KVQuant) and achieves a QK-multiply kernel speedup of up to 1.27× over FP16 matrix multiplication, with no statistically significant loss in LLM downstream task accuracy (Wu et al., 1 Feb 2025).
• For normalized attention, QKNorm delivers an average of +0.928 BLEU improvement in low-resource translation, with ablations confirming stability and expressivity benefits due to bounded similarity computation and adaptive scaling (Henry et al., 2020).

5. Connections and Broader Design Patterns

GIQT exemplifies a broader paradigm in the design of attention mechanisms:

• Modifying the geometry of query/key spaces—via rotation, projection, normalization, or learned transformations—enables both expressivity and increased robustness to distributional shifts or quantization constraints.
• Geometric transformations may be strictly data-driven (via learned MLPs on side-information), analytically determined (e.g., polar decompositions), or inductively biased (e.g., spherical, hyperbolic, or anisotropic metrics).
• The QKNorm approach suggests a general pattern: mapping query/keys to the desired Riemannian manifold, employing normalization or canonical parameterization, and learning global or head-wise scaling parameters to adapt similarity distribution for downstream pooling (Henry et al., 2020).

A plausible implication is that future GIQT variants may extend to more complex, non-Euclidean geometry (e.g., hyperbolic, product spaces) or adaptive metrics, and introduce further model-data alignment via geometry-aware conditioning.

6. Practical Considerations and Computational Overheads

GIQT modules are engineered for minimal runtime and memory overhead:

Implementation	Core Overhead	Parameters Added	Empirical Speedup/Cost
GIQT (vision) (Hambarde et al., 29 Jan 2026)	$<3\%$ decoder FLOPs	$O(10^5)$ ($(4d_g)d r$)	+0.64–3.75% R1, +1.16–4.42% mAP
PolarQuant (Wu et al., 1 Feb 2025)	$<5$ bits/dim; small lookup table	Per-channel $2^m$ floats	1.27× QK-mult acceleration
QKNorm (Henry et al., 2020)	Single scalar scale ($g$)	1	+0.928 BLEU avg.

Parameters and additional FLOPs are negligible compared to backbone size or global attention cost, and all methods avoid altering or slowing baseline transformer attention architectures.

7. Outlook and Research Trajectories

Geometry-Induced Query-Key Transformation represents a convergent trajectory in both practical LLM inference and robust cross-view retrieval. Extending the family of GIQT techniques may involve:

• Embedding complex, domain-informed geometric metadata within the similarity computation kernel.
• Employing learnable non-Euclidean metrics for data with hierarchical, graph, or spatial structure.
• Exploring meta-learned or data-driven adaptation of similarity geometry at run-time.

A plausible implication is that attention mechanisms will increasingly leverage explicit geometric priors, tailored to the invariances and structure of specific domains, to reconcile both computational and representational constraints.