Hyperbolic Cross-Attention Mechanism

• Hyperbolic cross-attention mechanism is a method that applies hyperbolic geometry to cross-attention by replacing Euclidean similarity with curvature-aware operations to capture hierarchical structures.
• It leverages models like the Poincaré ball and hyperboloid to perform distance computations and Möbius aggregation, effectively supporting tree-like data with exponential volume growth.
• The approach improves model efficiency and performance in tasks such as neural machine translation, graph modeling, and multimodal fusion by preserving geometric consistency.

A hyperbolic cross-attention mechanism is a generalization of standard Transformer-style cross-attention that replaces the conventional Euclidean space similarity and aggregation functions with curvature-aware operations derived from hyperbolic geometry. This approach is motivated by the exponential volume growth and natural support for tree-like or hierarchical structures provided by hyperbolic spaces, resulting in improved modeling of data where hierarchy or latent tree-structure plays a fundamental role, including language, graphs, and multimodal fusion tasks (Tseng et al., 2023, Gulcehre et al., 2018, Choudhury et al., 25 Jan 2026). Typical instantiations employ the Poincaré ball or hyperboloid models, along with Möbius or Lorentzian operations, and adapt attention aggregation and fusion to respect geodesic structure and the algebra of negative curvature.

1. Mathematical Foundations of Hyperbolic Geometry

Hyperbolic cross-attention is built on hyperbolic manifolds—spaces of constant negative sectional curvature—most commonly realized via the Poincaré ball, Poincaré half-space, or hyperboloid models.

• Poincaré Ball Model:

The $d$-dimensional Poincaré ball of curvature $-c$ ($c>0$) is $$\mathbb{B}^d_c = \{ x \in \mathbb{R}^d : \| x \| < 1/\sqrt{c}\}$$, with Riemannian metric $g_x(u,v) = \lambda_x^2 \langle u,v \rangle_2$, scaling factor $\lambda_x = 2 / (1 - c \| x \|^2)$, and geodesic (hyperbolic) distance

$$d_c(x, y) = \frac{1}{\sqrt{c}}\operatorname{arcosh}\left(1 + 2c\frac{\|x-y\|^2}{(1-c\|x\|^2)(1-c\|y\|^2)} \right).$$

Core Möbius operations include addition $\oplus_c$, scalar multiplication $\otimes_c$, and exponential/logarithmic maps $\exp^c_x$, $\log^c_x$ (Choudhury et al., 25 Jan 2026).

• Hyperboloid and Klein Models:

The $n$-dimensional hyperboloid model is embedded in $\mathbb{R}^{n+1}$, subject to the Minkowski metric $$\langle \mathbf u, \mathbf v \rangle_M = \sum_{i=1}^n u_iv_i - u_{n+1} v_{n+1}$$. Geodesics and distance computation rely on this indefinite inner product:

$d_{\mathbb{H}}(\mathbf q,\mathbf k) = \arccosh(-\langle \mathbf q, \mathbf k \rangle_M).$

Points are mapped between the hyperboloid and Klein models for efficient aggregation and projection (Gulcehre et al., 2018).

• Poincaré Half-Space & Entailment Cones:

The Poincaré upper half-space $\mathcal{H}^d = \{ x \in \mathbb{R}^d : x_d > 0\}$, with metric $g_x = g_e / x_d^2$, enables the construction of "shadow cones" for encoding entailment and hierarchy (Tseng et al., 2023).

These geometric frameworks support key algebraic and distance computations required for similarity, attention weighting, and aggregation.

2. Hyperbolic Cross-Attention Mechanism: Formulation

Hyperbolic cross-attention mechanisms replace the conventional query-key similarity and value aggregation with their geometry-aware analogs.

1. Euclidean-to-Hyperbolic Projection: Each modality or layer output, typically a Euclidean vector $h \in \mathbb{R}^d$, is projected to hyperbolic space using the exponential map at the origin:

$\mathcal{H}_i = \exp^c_0(h_i)$

For example, in the Poincaré ball, this is

$$\exp^c_0(v) = \tanh(\sqrt{c}\|v\|)\frac{v}{\sqrt{c}\|v\|}$$

(Choudhury et al., 25 Jan 2026).

1. Hyperbolic Query, Key, Value Construction: Euclidean queries, keys, and values undergo linear projection as usual, with parameters $W_Q, W_K, W_V$. These are then mapped into hyperbolic space via the chosen exponential map (Gulcehre et al., 2018, Tseng et al., 2023, Choudhury et al., 25 Jan 2026).
2. Curvature-Aware Similarity: The cross-attention score for query $\mathcal{Q}_i$ and key $\mathcal{K}_j$ is the negative hyperbolic geodesic distance,

$s_{ij} = -d_c(\mathcal{Q}_i, \mathcal{K}_j)$

Some variants employ temperature scaling or additive bias (Gulcehre et al., 2018).

1. Hyperbolic Softmax and Attention Weights: The attention weights are computed using the standard softmax, but over curvature-aware scores:

   $$\alpha_{ij} = \frac{\exp(-d_c(\mathcal{Q}_i, \mathcal{K}_j))}{\sum_{j'} \exp(-d_c(\mathcal{Q}_i, \mathcal{K}_{j'}))}$$

   (Choudhury et al., 25 Jan 2026).

2. Hyperbolic Aggregation: Value aggregation is performed using Möbius-weighted sums in the Poincaré ball:

   $$\mathcal{O}_i = \bigoplus_{j=1}^n \left( \alpha_{ij} \otimes_c \mathcal{V}_j \right)$$

   or, in the hyperboloid setting, using the Einstein midpoint via Klein projection and Lorentz-factor reweighting, then projecting back (Gulcehre et al., 2018). In cone attention, aggregation remains Euclidean post attention-weight computation (Tseng et al., 2023).

3. Projection Back to Euclidean: The aggregated hyperbolic vector is mapped back to Euclidean space using the logarithmic map at the origin and fed into downstream layers.

The following table summarizes geometric operations employed:

Operation	Poincaré Ball	Hyperboloid/Klein	Poincaré Half-Space
Similarity	$-d_c(x, y)$	$-d_{\mathbb{H}}$	$- \sup_2(q, k)_d$
Aggregation	Möbius sum $\oplus_c$	Einstein midpoint	Euclidean weighted sum
Exp/Log mapping	$\exp^c_0$, $\log^c_0$	$\Exp_o$, $\Log_o$	Pseudo-exponential maps

3. Hierarchy-Aware Attention via Cone Mechanisms

Cone attention [Editor's term] is a hyperbolic cross-attention variant designed to encode hierarchies explicitly through entailment cones in the Poincaré half-space (Tseng et al., 2023).

• Entailment Cones:

Each point $u \in \mathcal{H}^d$ defines a cone—a region containing all descendants in a partial order induced by a light source (horosphere at height $h$ for "penumbral", or point at infinity for "umbral").

• Lowest Common Ancestor (LCA):

For points $u, v$, their LCA $\sup_2(u, v)$ is the root of the minimal cone containing both. The vertical coordinate $\sup_2(u, v)_d$ encodes shared hierarchy depth; this directly yields the cross-attention similarity:

$K(u, v) = \exp(-\gamma \cdot \sup_2(u, v)_d)$

with distinct closed-form expressions for penumbral and umbral constructions.

• Pipeline:

Linear projected tokens are mapped by a numerically stable pseudo-exponential, similarity matrices $S_{ij}$ are computed via negative LCA-depth, then standard softmax and Euclidean aggregation follow as in Transformers.

Cone attention is a drop-in replacement for dot-product attention with the principal difference being hierarchy-aware similarity, enabling parameter-efficient encoding of tree-structured relations (Tseng et al., 2023).

4. Distinctions from Euclidean Cross-Attention

Euclidean cross-attention operates via

• Dot-product similarity: $\langle Q, K \rangle / \sqrt{d}$
• Softmax over scores
• Euclidean weighted sum: $O_i = \sum_j \alpha_{ij} V_j$

Hyperbolic cross-attention replaces:

• Inner product with curvature-aware distance ($d_c$ or $d_{\mathbb{H}}$) or LCA depth
• Sum with Möbius or Einstein aggregation schemes, respecting hyperbolic convexity
• All projections and normalizations are performed using the appropriate mappings between Euclidean and hyperbolic spaces, with full geometric consistency retained throughout (Gulcehre et al., 2018, Choudhury et al., 25 Jan 2026).

This shift ensures that geometric relationships with exponential expansion and partially ordered structure are preserved, enabling low-distortion embedding and aggregation of data with hierarchy or power-law connectivity.

5. Implementation and Pseudocode

A typical implementation pipeline:

1. Project Euclidean features (e.g., audio $H^{(a)}$, visual $H^{(v)}$) via $\exp^c_0$.
2. Compute hyperbolic queries, keys, and values after linear transformation.
3. Compute negative hyperbolic distances as similarity scores.
4. Apply softmax normalization for attention weights.
5. Aggregate values via Möbius sum or Einstein midpoint.
6. Map back to Euclidean space via log map for further processing.

Example pseudocode (Poincaré ball version, as in FOCA (Choudhury et al., 25 Jan 2026)):

for i in range(n): q_a[i] = exp_c0(W_Q @ H_a[i]) k_v[i] = exp_c0(W_K @ H_v[i]) v_v[i] = exp_c0(W_V @ H_v[i]) for i in range(n): for j in range(n): s_a2v[i, j] = -d_c(q_a[i], k_v[j]) alpha_a2v[i, :] = softmax(s_a2v[i, :]) for i in range(n): O_a2v[i] = mobius_sum([alpha_a2v[i, j] * v_v[j] for j in range(n)]) O[i] = log_c0(O_a2v[i])

Here, $\exp^c_0$, $d_c$, $\oplus_c$, and $\log^c_0$ are defined as above.

6. Practical Applications, Empirical Behavior, and Curvature Control

Empirical work demonstrates gains of hyperbolic and cone cross-attention mechanisms in contexts where data exhibits latent hierarchy:

• Language and Graph Modeling:

Hyperbolic cross-attention yields improvements in neural machine translation, graph attention, and language modeling, notably reducing required dimensionality for a given performance level (Tseng et al., 2023, Gulcehre et al., 2018).

• Multimodal Fusion:

In malware classification (FOCA), hyperbolic cross-attention between audio and visual representations, with explicit curvature-aware dependencies and Möbius-based fusion, outperforms both unimodal and Euclidean multimodal models, showing improved alignment of hierarchical features (Choudhury et al., 25 Jan 2026).

• Curvature Parameter ($c$):

The curvature $c$ governs the degree of “tree-likeness”. Larger values induce stronger hierarchical separation, while $c \to 0$ recovers Euclidean geometry. Training may fix $c$ or treat it as a learnable parameter to adapt to task geometry.

• Computational Overhead:

Commonly, additional computational cost compared to dot-product attention is limited (10–20% in CUDA implementations). Modern deep learning frameworks support the vectorized operations required for Möbius and hyperbolic geometry calculations (Tseng et al., 2023).

7. Model Size Efficiency and Performance Characteristics

Hyperbolic cross-attention mechanisms, particularly cone attention, exhibit improved representation efficiency:

• On IWSLT’14 De–En NMT, cone attention at $d=16$ matches dot-product attention at $d=128$.
• For vision (DeiT-Ti), penumbral cone attention with $d=16$ outperforms dot-product attention at $d=64$.
• Empirical results show consistent task-level improvements when latent community or hierarchy is present (Tseng et al., 2023).

A plausible implication is that enforcing hyperbolic structure acts as an architectural regularizer, driving compactness of attention representations while preserving expressivity for hierarchical semantics.

---

References:

• "Coneheads: Hierarchy Aware Attention" (Tseng et al., 2023)
• "FOCA: Multimodal Malware Classification via Hyperbolic Cross-Attention" (Choudhury et al., 25 Jan 2026)
• "Hyperbolic Attention Networks" (Gulcehre et al., 2018)