Conjecture: RoPE structures keys and queries to enable better retrieval

Establish whether Rotary Position Embeddings (RoPE) structure the key and query vectors in Transformer attention mechanisms in a manner that conditions them for improved retrieval performance compared to the absence of RoPE (NoPE).

Background

In the ablations section, the authors compare attention performance with and without Rotary Position Embeddings (RoPE). They observe that removing RoPE increases the loss difference between LUCID and standard attention in long-context settings and hypothesize that RoPE may organize the key and query representations to facilitate retrieval.

Based on these empirical observations, the authors explicitly conjecture a causal role for RoPE in structuring the representations, suggesting that this effect could enhance retrieval ability. This conjecture invites a formal investigation to confirm or refute the hypothesized mechanism.