Directional Rotary Position Embedding (DRoPE)

• DRoPE is a position encoding scheme that extends rotary embeddings to accurately capture angular periodicity and directional information for agent interactions.
• It modifies RoPE by applying uniform rotations on embedding blocks, ensuring attention scores depend solely on heading differences modulo 2π.
• Empirical evaluations in trajectory forecasting demonstrate DRoPE’s improved accuracy and efficiency compared to conventional query-centric relative position embeddings.

Directional Rotary Position Embedding (DRoPE) is a position encoding scheme designed for efficient modeling of agent interactions involving both position and heading (directional) information within transformer architectures. DRoPE extends the Rotary Position Embedding (RoPE) mechanism to address inherent limitations in handling angular periodicity, critical for domains such as multi-agent trajectory forecasting where agent heading must be captured with exact periodicity and relative orientation while avoiding the memory inefficiency of explicit relative position embeddings (Zhao et al., 19 Mar 2025).

1. Motivation and Challenges in Encoding Agent Interactions

Traditional approaches to agent interaction modeling in trajectory generation can be categorized as scene-centric, agent-centric, or query-centric, each facing trade-offs among prediction accuracy, computational complexity, and space efficiency:

• Scene-centric frameworks use absolute coordinates, minimizing computational and memory costs (space: $O(NH(2d_k+d_v))$, time: $O(N^2H d_k)$) but result in degraded accuracy as agent interactions are not directly modeled as relative transformations.
• Agent-centric approaches normalize coordinates per agent, attaining high accuracy but incurring $O(N^2)$ time complexity for $N$ agents.
• Query-centric paradigms, particularly those employing explicit Relative Position Embeddings (RPE), capture pairwise spatial relationships accurately but suffer from $O(N^2)$ memory complexity due to the need to store per-pair embeddings (Zhao et al., 19 Mar 2025).

RoPE, originally developed for sequential data, encodes relative positions implicitly through block-diagonal 2D rotations on subspaces of the embedding vectors, maintaining $O(N)$ memory complexity. However, its design for linear positional encodings with multiple frequency scales $\theta_\ell$ fails to preserve true angular periodicity—specifically, its rotary transformation does not respect the $\phi \to \phi + 2\pi$ equivalence necessary for modeling agent headings (Zhao et al., 19 Mar 2025). DRoPE was introduced to circumvent this failure while retaining the computational benefits of RoPE.

2. Mathematical Formulation of DRoPE

In standard RoPE, given vector $X\in \mathbb{R}^{2d_k}$ at position $m$: $$f^{\rightarrow}(X, m) = \operatorname{BlockDiag}(R(m\theta_0), ..., R(m\theta_{d_k-1}))\cdot X,$$ with each $2\times2$ block

$$R(\phi) = \begin{pmatrix} \cos\phi & -\sin\phi \ \sin\phi & \cos\phi \end{pmatrix},$$

and $\theta_\ell = 10000^{-\ell/d_k}$. Here, the dot product between two such rotated vectors implicitly depends only on the relative offset $m_i - m_j$, allowing RoPE to represent relative positional information efficiently.

DRoPE modifies this by setting all rotary scales equal (i.e., $\theta_\ell \equiv 1$ for all blocks) and applying rotations by heading angle $\phi \in [0, 2\pi)$: $$f^\angle(X, \phi) = \operatorname{BlockDiag}(R(\phi), ..., R(\phi))\cdot X.$$ This leads to the key property: $$\langle f^\angle(Q_i, \phi_i), f^\angle(K_j, \phi_j)\rangle = Q_i^\top \operatorname{BlockDiag}(R(\phi_j - \phi_i \bmod 2\pi), ..., R(\phi_j - \phi_i \bmod 2\pi)) K_j,$$ which ensures that the attention score depends only on the difference in heading modulo $2\pi$ (Zhao et al., 19 Mar 2025).

3. Correctness, Efficiency, and Complexity Analysis

DRoPE satisfies the general Relative Position Embedding property: for i, j, the attention score is a function only of $(\phi_i - \phi_j \bmod 2\pi)$, making it suitable for tasks involving periodic variables such as orientation.

Space and Time Complexity

Method	Space Complexity	Time Complexity
Scene-centric	$O(NH(2d_k+d_v))$	$O(N^2H d_k)$
Agent-centric	$O(NH(2d_k+d_v))$	$O(N^2H d_k)$ (via repetition)
Query-centric + RPE	$O(N^2H(d_k + d_v))$	$\ge O(N^2H d_k) + O(N^2$ MLP)
Vanilla RoPE	$O(NH(2d_k+d_v))$	$O(N^2H d_k)$
DRoPE (pos. + ang.)	$O(NH(2d_k+d_v))$	$O(N^2H d_k)$

DRoPE achieves exact angular relative encoding with the same asymptotic time and space costs as scene-centric or vanilla-RoPE methods, unlike query-centric RPE methods, whose memory requirements grow quadratically with $N$ (Zhao et al., 19 Mar 2025).

4. Empirical Evaluation

Extensive evaluation on the Waymo Motion Dataset v1.2 in closed-loop simulation settings confirms the practical effectiveness of DRoPE. Using DRoPE-Traj (3M parameters, query-centric structure):

• minADE: 1.2626 (lower is better)
• REALISM: 0.7625 (higher is better)

Compared to baselines such as UniMM, SMART, and BehaviorGPT, DRoPE-Traj outperforms all on trajectory accuracy at comparable or lower memory and computational cost (Zhao et al., 19 Mar 2025). For increasing latent dimension $d_k$, DRoPE maintains memory usage close to scene-centric baselines, while RPE-based methods see rapid increases in evaluation memory and FLOPs (4–6$\times$ higher for RPE).

Integration Strategies

Ablation studies reveal that head-by-head DRoPE–RoPE integration (dedicated heads for position and heading) yields the best trade-off between accuracy and efficiency, outperforming intra-head approaches which split dimensions within a head (Zhao et al., 19 Mar 2025).

5. Theoretical Extensions and Lie-Algebraic Foundations

A systematic mathematical framework for RoPE and DRoPE has been developed using Lie algebra theory (Liu et al., 7 Apr 2025). In this framework, DRoPE arises by constructing block-diagonal generators in the special orthogonal Lie algebra so(2N), guaranteeing relativity: $$(R_{\rm DR}(p_1)q)^\top(R_{\rm DR}(p_2)k) = q^\top R_{\rm DR}(p_2 - p_1)k$$ and reversibility (injectivity) as long as the frequency schedule avoids global wrapping. Furthermore, DRoPE allows principled modeling of directional interactions between embedding blocks by introducing an orthogonal change of basis, which enables richer representations in higher dimensions (Liu et al., 7 Apr 2025).

6. Limitations, Best Practices, and Future Directions

Current DRoPE encodes only planar (2D) headings. Its extension to higher-dimensional orientation (roll, pitch) or multi-modal embeddings (including speed or other circular variables) is proposed as future work (Zhao et al., 19 Mar 2025). DRoPE maintains $O(N^2)$ time complexity with respect to the number of agents; integrating DRoPE with sparse or locality-attention mechanisms is expected to improve scalability in high-density scenarios.

Best practices indicate that dedicating individual attention heads to DRoPE or RoPE (head-by-head integration) is preferable to mixing features within a head. Implementation minimally alters transformer architectures: only the rotary block construction is changed in the relevant heads or subspaces; no changes to the transformer core or sequence of operations are required (Zhao et al., 19 Mar 2025).

Possible future directions include adopting learned frequency schedules to balance angular periodicity and frequency-based positional encoding, applying DRoPE in areas with periodic variables (robotics, meteorology), and exploring richer parameterizations from the Lie-algebraic blueprint for N-dimensional position encoding (Liu et al., 7 Apr 2025).

7. Comparison to Related Approaches: Directionality and Decoupling of Content/Position

Recent work on Polar Coordinate Position Embeddings (PoPE) (Gopalakrishnan et al., 5 Sep 2025) and theoretical analyses of RoPE’s entanglement between content (“what”) and position (“where”) have shown that augmenting rotary mechanisms to decouple or directionally bias embeddings improves empirical performance on sequence modeling and length extrapolation. PoPE introduces explicit decoupling and can be extended toward directional encoding via asymmetric biases, conceptually related to per-frequency directional bias variants of DRoPE described in foundational work on rotary embeddings (Su et al., 2021, Gopalakrishnan et al., 5 Sep 2025). DRoPE can thus be viewed both as a minimal, principled periodic angular extension of RoPE and as a member of a broader class of directional, decoupling positional encoding schemes.

---

References

(Zhao et al., 19 Mar 2025, Liu et al., 7 Apr 2025, Gopalakrishnan et al., 5 Sep 2025, Su et al., 2021)