Circle-RoPE: Geometric Decoupling in LVLMs

• Geometric decoupling via Circle-RoPE is a method that redefines positional embeddings by aligning image tokens on a circular manifold orthogonal to text tokens.
• It maps image token positions to a fixed-radius plane, ensuring uniform Euclidean distances from text tokens and reducing artificial cross-modal dependencies.
• Using Alternating Geometry Encoding, Circle-RoPE demonstrates improved performance in vision-language tasks by balancing local spatial precision with global bias-free alignment.

Geometric decoupling in the context of positional encoding refers to the architectural disentanglement of position-related inductive biases along orthogonal or complementary axes in neural sequence or vision-LLMs. Circle-RoPE is a canonical embodiment of this principle within large vision-LLMs (LVLMs). By explicitly designing the relative geometry of text and image token indices, Circle-RoPE eliminates spurious cross-modal positional dependencies inherent in standard extensions of Rotary Position Embedding (RoPE), thereby enabling more robust and unbiased multimodal feature fusion (Wang et al., 22 May 2025).

1. Mathematical Foundations of Rotary Positional Embedding

Standard RoPE encodes relative position by rotating each even/odd pair of a $d$-dimensional token embedding $x\in\mathbb R^d$ by frequency-specific angles, parameterized as

$$\begin{pmatrix} x'_{2i} \ x'_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos(\theta_{i}\,p) & -\sin(\theta_{i}\,p) \ \sin(\theta_{i}\,p) & \cos(\theta_{i}\,p) \end{pmatrix} \begin{pmatrix} x_{2i} \ x_{2i+1} \end{pmatrix}$$

with base frequencies $\theta_{i}=1/10000^{2i/d}$ and $p$ the token position. This structure ensures that the self-attention kernel $\langle Q'_p, K'_q\rangle$ encodes $\cos[\theta_i(p-q)]$, i.e., explicit relative distance dependence. Multimodal RoPE variants (e.g., M-RoPE) attempt to extend this principle to 2D grids for image patches by applying two orthogonal RoPE transformations for each spatial axis and then linearizing the sequence to concatenate with text tokens.

2. Motivation: Spurious Cross-Modal Biases in Multimodal RoPE

When text and image tokens are indexed consecutively, conventional RoPE and its multimodal extensions inadvertently enforce relative positional dependencies between the text index and every image patch index. This coupling introduces positional biases that manifest as spurious alignments—for example, two image patches with the same semantic content but at different spatial grid positions receive distinct position codes, leading to inconsistent text-image associations. Such biases are empirically implicated in degraded cross-modal reasoning and attention in LVLMs (Wang et al., 22 May 2025).

3. Circle-RoPE: Cone-Like Orthogonal Decoupling

Circle-RoPE re-parameterizes the position indices of image tokens so that, in the extended positional embedding space, all image tokens lie on a circle within a plane orthogonal to the text token "axis." The mapping proceeds via:

• Centralizing the spatial grid of image tokens,
• Compounding each image token's angle as a convex combination ($\alpha$) of its spatial-origin angle (computed via $\operatorname{atan2}$) and its grid-sequence angle,
• Assigning a radius $R$, typically fixed or scaled by maximal grid norm,
• Projecting circle coordinates into the plane orthogonal to the (normalized) text axis vector $x\in\mathbb R^d$0 through an orthonormal basis $x\in\mathbb R^d$1,
• Optionally fusing back a proportion $x\in\mathbb R^d$2 of the planar coordinates to recover spatial layout (termed Decoupled Fusion Factor, DFF).

Formally, each image coordinate $x\in\mathbb R^d$3 is mapped to

$x\in\mathbb R^d$4

with $x\in\mathbb R^d$5 the projection onto the orthogonal plane and $x\in\mathbb R^d$6 the centralized grid.

As a result, text token indices remain aligned with the 1D axis, while all image patches attain an identical radial displacement in a transverse plane. The geometric consequence is that, for every text token $x\in\mathbb R^d$7 and image token $x\in\mathbb R^d$8,

$x\in\mathbb R^d$9

which is constant for all $$\begin{pmatrix} x'_{2i} \ x'_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos(\theta_{i}\,p) & -\sin(\theta_{i}\,p) \ \sin(\theta_{i}\,p) & \cos(\theta_{i}\,p) \end{pmatrix} \begin{pmatrix} x_{2i} \ x_{2i+1} \end{pmatrix}$$0 given $$\begin{pmatrix} x'_{2i} \ x'_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos(\theta_{i}\,p) & -\sin(\theta_{i}\,p) \ \sin(\theta_{i}\,p) & \cos(\theta_{i}\,p) \end{pmatrix} \begin{pmatrix} x_{2i} \ x_{2i+1} \end{pmatrix}$$1.

4. Quantifying Decoupling: Per-Token Distance Metric

To measure positional independence across modalities, the Per-Token Distance (PTD) metric is introduced. For text token $$\begin{pmatrix} x'_{2i} \ x'_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos(\theta_{i}\,p) & -\sin(\theta_{i}\,p) \ \sin(\theta_{i}\,p) & \cos(\theta_{i}\,p) \end{pmatrix} \begin{pmatrix} x_{2i} \ x_{2i+1} \end{pmatrix}$$2 and image token $$\begin{pmatrix} x'_{2i} \ x'_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos(\theta_{i}\,p) & -\sin(\theta_{i}\,p) \ \sin(\theta_{i}\,p) & \cos(\theta_{i}\,p) \end{pmatrix} \begin{pmatrix} x_{2i} \ x_{2i+1} \end{pmatrix}$$3, define

$$\begin{pmatrix} x'_{2i} \ x'_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos(\theta_{i}\,p) & -\sin(\theta_{i}\,p) \ \sin(\theta_{i}\,p) & \cos(\theta_{i}\,p) \end{pmatrix} \begin{pmatrix} x_{2i} \ x_{2i+1} \end{pmatrix}$$4

and for all image tokens, compute

$$\begin{pmatrix} x'_{2i} \ x'_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos(\theta_{i}\,p) & -\sin(\theta_{i}\,p) \ \sin(\theta_{i}\,p) & \cos(\theta_{i}\,p) \end{pmatrix} \begin{pmatrix} x_{2i} \ x_{2i+1} \end{pmatrix}$$5

The global PTD is

$$\begin{pmatrix} x'_{2i} \ x'_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos(\theta_{i}\,p) & -\sin(\theta_{i}\,p) \ \sin(\theta_{i}\,p) & \cos(\theta_{i}\,p) \end{pmatrix} \begin{pmatrix} x_{2i} \ x_{2i+1} \end{pmatrix}$$6

A PTD of zero demonstrates perfect geometric decoupling: every text token is equidistant from every image token in the positional index space, thereby guaranteeing absence of artificial bias from the positional embedding stage. Empirically, Circle-RoPE achieves $$\begin{pmatrix} x'_{2i} \ x'_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos(\theta_{i}\,p) & -\sin(\theta_{i}\,p) \ \sin(\theta_{i}\,p) & \cos(\theta_{i}\,p) \end{pmatrix} \begin{pmatrix} x_{2i} \ x_{2i+1} \end{pmatrix}$$7 (excepting the optional $$\begin{pmatrix} x'_{2i} \ x'_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos(\theta_{i}\,p) & -\sin(\theta_{i}\,p) \ \sin(\theta_{i}\,p) & \cos(\theta_{i}\,p) \end{pmatrix} \begin{pmatrix} x_{2i} \ x_{2i+1} \end{pmatrix}$$8-fusion for spatial retention), in contrast to nontrivial PTD for standard multimodal RoPE (Wang et al., 22 May 2025).

5. Reduction of Cross-Modal Bias Through Orthogonality

With Circle-RoPE, the underlying positional attention bias between text and image tokens becomes invariant: $$\begin{pmatrix} x'_{2i} \ x'_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos(\theta_{i}\,p) & -\sin(\theta_{i}\,p) \ \sin(\theta_{i}\,p) & \cos(\theta_{i}\,p) \end{pmatrix} \begin{pmatrix} x_{2i} \ x_{2i+1} \end{pmatrix}$$9 is constant in $\theta_{i}=1/10000^{2i/d}$0 for fixed $\theta_{i}=1/10000^{2i/d}$1. Thus, text-to-image attention is not modulated by their sequential or spatial offset, but only by semantic content. Only intra-modal relative positions (text-text or image-image) retain their distinct encoded structure. This geometric construction disables the formation of spurious cross-modal alignments that would otherwise arise via RoPE extension to joint indices.

6. Staggered Layer Alternation: Exploiting Complementary Geometries

While Circle-RoPE delivers global positional decoupling, local spatial precision may be degraded relative to strongly planar RoPE variants (e.g., M-RoPE), which preserve gridwise translation. To balance these objectives, the Alternating Geometry Encoding (AGE) strategy is deployed: odd-numbered Transformer layers use standard M-RoPE, while even-numbered layers employ Circle-RoPE (with circular image indices and DFF mixing). This configuration leverages the high-fidelity local geometry of M-RoPE at shallow layers and the bias-free cross-modal global fusion of Circle-RoPE at deeper layers or in alternate blocks. Empirically, strict alternation yields superior performance to pure or partially staggered approaches (Wang et al., 22 May 2025).

RoPE Variant	Cross-Modal Bias	Spatial Precision	Typical Layer
M-RoPE	High	High	Odd
Circle-RoPE	Zero	Moderate (w/ DFF)	Even

Ablation studies indicate optimal mixing coefficients at $\theta_{i}=1/10000^{2i/d}$2 for angular interpolation, radius $\theta_{i}=1/10000^{2i/d}$3, and DFF weight $\theta_{i}=1/10000^{2i/d}$4.

7. Empirical Evaluation and Implementation Practice

Circle-RoPE combined with the AGE pattern was evaluated on Qwen2.5-VL-3B fine-tuned atop a frozen vision encoder and multimodal projector using $\theta_{i}=1/10000^{2i/d}$51 M MAmmoTH-VL-Instruct examples. Results demonstrate consistent improvement versus baseline M-RoPE only:

• Mean score across 10 vision-language tasks: 66.95 (M-RoPE) vs. 68.28 (Circle-RoPE + AGE).
• Notably, on MathVista: 62.4 vs. 63.4, MMStar: 54.13 vs. 58.20, AI2D: 78.14 vs. 81.80, etc.

Implementation recommendations:

• Precompute $\theta_{i}=1/10000^{2i/d}$6 per-image resolution before training or inference.
• Use hyperparameters $\theta_{i}=1/10000^{2i/d}$7, $\theta_{i}=1/10000^{2i/d}$8, $\theta_{i}=1/10000^{2i/d}$9; process text and image tokens identically except for positional index mapping.
• Only fine-tune the LLM, not the vision backbone or projector.

Code resources are released at https://github.com/lose4578/CircleRoPE (Wang et al., 22 May 2025).

8. Significance, Limitations, and Context

Circle-RoPE exemplifies geometric decoupling by orthogonalizing modality-specific position embeddings, demonstrating that architectural geometry controls cross-modal inductive bias. The PTD metric provides a quantitative gauge of this geometric independence. The DFF mechanism allows spatial layout retention despite decoupling. Staggered-layer integration validates that no single positional embedding geometry dominates across all stages of hierarchical multimodal inference.

A plausible implication is that analogous geometric decoupling strategies could yield benefits in other hybrid domains where spurious cross-modal distance biases affect learning, although detailed verification resides in future studies. Current empirical results restrain claims of superiority to models with the specific training regimes and LLM backbones evaluated.

In summary, geometric decoupling, as realized in Circle-RoPE, marks a principled and empirically supported advance in position-encoding strategies for LVLMs, enabling unbiased cross-modal context modeling and robust spatial reasoning (Wang et al., 22 May 2025).