CARoPE: Context-Aware Rotary Embedding

• The paper introduces CARoPE, a dynamic rotary embedding mechanism that adapts attention-head frequencies based on token embeddings.
• CARoPE replaces static sinusoidal frequencies with token-dependent ones via a lightweight neural network to enhance positional encoding.
• Empirical results demonstrate significant perplexity reduction and throughput improvements, especially for longer context lengths.

Context-Aware Rotary Positional Embedding (CARoPE) is a generalization of Rotary Positional Embedding (RoPE) designed to endow Transformer models with token- and context-sensitive positional representations while retaining the computational and architectural advantages of RoPE. Unlike conventional RoPE, which imposes a static, input-agnostic frequency structure, CARoPE dynamically generates attention-head-specific frequency patterns conditioned on token embeddings. This enables richer relative position encoding, improved extrapolation to longer contexts, and enhanced expressivity in both language and multimodal settings (Veisi et al., 30 Jul 2025, Chen et al., 18 May 2025).

1. From Static to Context-Aware Rotary Embeddings

Standard RoPE applies a fixed rotation in the complex plane to each attention head’s query and key subspaces, parameterized by pre-defined, position-dependent sinusoidal frequencies. These are agnostic to both token identity and sequence context: each frequency $\theta_k$ is determined solely by its coordinate index, and all tokens at a given position are encoded identically regardless of content. As a result, RoPE lacks the flexibility to adapt to token- or context-specific relational patterns, limiting its effectiveness in tasks demanding strong context awareness or cross-modal alignment (Su et al., 2021, Chen et al., 2024).

CARoPE directly addresses this limitation by replacing RoPE’s fixed base frequencies with token- and head-dependent frequencies generated by a small neural network. This mechanism allows each attention head to modulate its positional encoding rate based on the input token embedding, achieving “context-awareness” and enabling different heads to learn distinct rotary speeds or phase-accumulation rates across the sequence (Veisi et al., 30 Jul 2025).

2. Mathematical Formulation

2.1 Standard RoPE

Given even model dimension $d$, RoPE splits vectors $v\in\mathbb{R}^d$ into $d/2$ coordinate pairs. For position $m$, the $k$-th rotary frequency is

$\theta_k = 10000^{-2k/d}.$

The accumulated phase is $\varphi_k(m) = m \cdot \theta_k$. Each 2-D subspace of $v$ is rotated by

$$R(\varphi_k(m)) = \begin{bmatrix} \cos \varphi_k(m) & -\sin \varphi_k(m) \ \sin \varphi_k(m) & \cos \varphi_k(m) \end{bmatrix}$$

so that

$$\text{RoPE}(v,m)_{2k:2k+1} = R(\varphi_k(m)) \cdot [v_{2k}; v_{2k+1}].$$

This leads to relative position encoding because the attention score depends only on $m-n$ for tokens at positions $m$, $n$.

2.2 CARoPE’s Contextual Phase

Let $x_t\in\mathbb{R}^d$ denote the embedding for token $t$ and $H$ the attention head count. For each head $h$, a scalar frequency $f_h(x_t) \in (0,1)$ is computed as

$$f(x_t) = \frac{1}{\operatorname{softplus}(x_t W) + 1} \in \mathbb{R}^H$$

where $W \in \mathbb{R}^{d \times H}$ is a learned projection and $\operatorname{softplus}(a) = \log(1 + e^{a})$. This makes the rotary frequency head- and token-dependent.

For head $h$ and index $k$, the “base frequency” is $f_h(x_t)$. The phase up to position $m$ is accumulated as

$\varphi_k^{(h)}(m) = \sum_{t=1}^m f_h(x_t)^k.$

As in RoPE, this phase modulates the rotary transformation on $(2k,2k+1)$ subspaces: $$\text{CARoPE}_h(v,m)_{2k:2k+1} = R(\varphi_k^{(h)}(m)) \cdot [v_{2k}; v_{2k+1}]$$ where $R$ is the standard $2\times 2$ rotation. All frequencies remain bounded, and if $f_h(x_t)$ is initialized to RoPE’s $\theta_k$, CARoPE reduces exactly to RoPE (Veisi et al., 30 Jul 2025).

3. Integration with Transformer Architectures

Within a standard multi-head attention block, CARoPE is applied after projection to queries $Q_h$ and keys $K_h$ but before computing attention logits. The process is:

1. For each head $h$ and each token embedding $x_t$, compute $f_h(x_t)$ via projection and non-linearity.
2. For each rotary dimension $k$ and position $m$, accumulate the phase $\varphi_k^{(h)}(m) = \sum_{t=1}^{m} f_h(x_t)^k$.
3. Apply the rotation $R(\varphi_k^{(h)}(m))$ to the $(2k,2k+1)$-slice of $Q_h$ and $K_h$.
4. Proceed with the standard dot-product attention.

Overhead is limited to a single $d \times H$ projection matrix $W$ per layer, introducing negligible parameter and computational cost compared to the $O(L^2 d)$ cost of the attention operation. Temporary buffers for the phases are small and can be streamed efficiently (Veisi et al., 30 Jul 2025).

4. Empirical Evaluation and Quantitative Results

Experimental Protocol

Experiments employed the FineWeb-Edu-10B corpus (1.3T tokens). Models tested:

• GPT-2-Tiny: 6 layers, $d=512$, $H=8$, 44M parameters.
• GPT-2-Small: 12 layers, $d=768$, $H=10$, 124M parameters.

Training used next-token prediction with $L=512$ context and batch sizes of 32–64, for 19K steps (approx. 1 epoch). Optimization leveraged Adam with standard schedules and hardware (dual NVIDIA H100 GPUs).

Perplexity and Throughput

Empirical results for validation perplexity (lower is better):

Model	L=512	L=1024
Sinusoidal	22.14	166.18
Learnable	21.90	–
RoPE	21.31	56.61
CARoPE	21.23	21.39

For GPT-2-Tiny, CARoPE reduces perplexity by over 60% relative to RoPE at $L=1024$; for GPT-2-Small, by over $2.5\times$ (Veisi et al., 30 Jul 2025). Training throughput improves (e.g., RoPE achieves $\sim$0.63M tok/s on Small, CARoPE $\sim$0.76M tok/s), attributed to enhanced numerical stability and improved GPU fusion.

No instability or convergence delays were observed for CARoPE relative to RoPE.

5. Efficiency and Scalability

CARoPE’s parameter and computational overhead is minimal:

• Parameter overhead: one $d\times H$ projection per layer.
• Compute: $O(dH)$ per token for the projection and $O(dLH)$ for phase accumulation; both are negligible compared to the $O(L^{2}d)$ softmax attention.
• Memory: overhead comprises the small $W$ matrix plus temporary $L \times (d/2H)$ buffers for the phases per head—these are minor compared to standard model and token storage.

Empirically, CARoPE matches or exceeds RoPE in speed due to numerically stable, input-bounded frequencies that facilitate GPU optimization. As $H$ is usually kept constant with model scaling, time and memory complexity matches RoPE ($O(L^{2}d)$ and $O(Ld)$, respectively) (Veisi et al., 30 Jul 2025).

6. Context-Aware Rotary Embeddings Beyond Language

In the multimodal domain, CARoPE has been deployed for multi-conditional image generation in architectures such as ContextAR. Here, each condition type (e.g., edges, text prompts) is provided with a standard 2D RoPE for spatial alignment, augmented with a learnable condition-specific positional embedding. This “CARoPE” fusion maintains both precise spatial correspondence and modality discrimination with only a minor parameter cost (one offset tensor $P_{k}$ per condition type). Ablation confirms measurable gains in output quality (e.g., improved FID and MSE relative to pure RoPE), demonstrating that context-aware position encoding principles extend naturally beyond sequence modeling to cross-modal tasks (Chen et al., 18 May 2025).

7. Relation to Other Context-Aware and Token-Dependent Positional Encodings

The core mechanism of CARoPE—conditioning frequencies or phase shifts on input tokens—substantiates a broader research trajectory. HoPE (High-frequency Rotary Positional Encoding) removes priors of monotonic long-term decay and proposes spectral filtering to retain positional encoding only in high-frequency bands, improving context awareness and extrapolation (Chen et al., 2024). Token-Aware Phase Attention (TAPA) further generalizes token-dependent phase modulation, using a learnable function of the token pair to define rotation, provably mitigating RoPE’s intrinsic distance bias and preserving variance for long-range attention (Yu et al., 16 Sep 2025). Both works inform CARoPE’s design strategies: privileging data-driven, content-aware phase shifts, and carefully managing spectral content to avoid extrapolation failures.

A key design challenge throughout these works is balancing learnability and stability: any context-aware phase function must preserve rotation orthogonality to avoid norm explosion, and must not reintroduce absolute positional biases that undermine the relative-only property of standard RoPE (Su et al., 2021, Veisi et al., 30 Jul 2025).

---

References:

• "Context-aware Rotary Position Embedding" (Veisi et al., 30 Jul 2025)
• "RoFormer: Enhanced Transformer with Rotary Position Embedding" (Su et al., 2021)
• "HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation" (Chen et al., 2024)
• "Context-Aware Autoregressive Models for Multi-Conditional Image Generation" (Chen et al., 18 May 2025)
• "Positional Encoding via Token-Aware Phase Attention" (Yu et al., 16 Sep 2025)