Demystifying the Slash Pattern in Attention: The Role of RoPE

Abstract: LLMs often exhibit slash attention patterns, where attention scores concentrate along the $Δ$-th sub-diagonal for some offset $Δ$. These patterns play a key role in passing information across tokens. But why do they emerge? In this paper, we demystify the emergence of these Slash-Dominant Heads (SDHs) from both empirical and theoretical perspectives. First, by analyzing open-source LLMs, we find that SDHs are intrinsic to models and generalize to out-of-distribution prompts. To explain the intrinsic emergence, we analyze the queries, keys, and Rotary Position Embedding (RoPE), which jointly determine attention scores. Our empirical analysis reveals two characteristic conditions of SDHs: (1) Queries and keys are almost rank-one, and (2) RoPE is dominated by medium- and high-frequency components. Under these conditions, queries and keys are nearly identical across tokens, and interactions between medium- and high-frequency components of RoPE give rise to SDHs. Beyond empirical evidence, we theoretically show that these conditions are sufficient to ensure the emergence of SDHs by formalizing them as our modeling assumptions. Particularly, we analyze the training dynamics of a shallow Transformer equipped with RoPE under these conditions, and prove that models trained via gradient descent exhibit SDHs. The SDHs generalize to out-of-distribution prompts.

• The paper provides a mechanistic explanation for slash-dominant attention patterns by linking low-rank pre-RoPE queries/keys and frequency interference.
• It employs rigorous empirical evaluations across models like Gemma-7B and Qwen2.5, demonstrating consistent out-of-distribution generalization.
• The findings suggest promising strategies for model compression and architectural improvements by leveraging inherent rank constraints in attention modules.

Demystifying the Slash Pattern in Attention: The Role of RoPE

---

Introduction

"Demystifying the Slash Pattern in Attention: The Role of RoPE" (2601.08297) presents an incisive mechanistic and mathematical account of the widespread "slash-dominant" patterns in attention matrices of LLMs that use Rotary Position Embedding (RoPE). The paper dissects both empirical and theoretical facets of how and why these Slash-Dominant Heads (SDHs)—where attention mass is concentrated along sub-diagonals corresponding to a fixed offset—arise robustly and generalize across input distributions and tasks. The results reveal that SDHs are an intrinsic architectural phenomena, directly attributable to the geometric and spectral properties of pre-RoPE queries and keys, as well as the nature of the RoPE frequency spectrum.

---

Empirical Characterization of SDHs

The investigation begins with a rigorous empirical assessment of SDHs across SOTA open-source RoPE-based LLMs, including Gemma-7B, Qwen2.5-7B-Instruct, and Llama3-8B-Instruct. The authors utilize a mathematical operationalization of SDHs: for a given attention head, slash-dominance at lag $\Delta$ is defined by the average attention score concentrated on the $\Delta$-th sub-diagonal surpassing a specific threshold. This is systematically computed using LongBenchV2 workloads and i.i.d. token prompts for out-of-distribution (OOD) assessment.

The results indicate:

• Pervasiveness across heads and models: SDHs with small $\Delta$ (e.g., $\Delta \in \{0,1,2,3,4\}$) are found throughout the networks of all studied models. Attention maps display canonical slash patterns both for short-range and extreme long-range offsets.
• OOD generalization: Empirically, the slash-dominant pattern persists even for OOD random-token prompts, with the average slash score either matching or exceeding those observed for in-distribution prompts (Figure 1, Figure 2).

Figure 1: Attention matrices in Qwen2.5-7B-Instruct exhibit strong sub-diagonal concentration for small and large $\Delta$, invariant to prompt distribution.

---

Mechanistic Attribution: Rank-One Structure and RoPE

A salient finding is that in SDHs, the pre-RoPE queries and keys are nearly always low-rank, approaching rank one. This phenomenon is robust across architectures and layers.

• Pre-PE queries and keys are token-invariant: Visualization (see Figure 3) shows very minor variation between different token positions within each SDH, i.e., each query/key is approximately a copy of the same direction vector.
• The cause is architectural, not input-dependent: Neither the hidden state representations themselves, nor the projection matrices $W_Q$, $W_K$, are inherently low-rank. The rank-one structure arises from their interaction, aided by token embeddings clustering around a cone (i.e., all embeddings having high cosine similarity with a common vector).
• Role of explicit biases (Qwen2.5): In Qwen2.5-7B-Instruct, large biases in projection layers further enhance the tendency toward rank-one pre-RoPE queries/keys.

Figure 3: Pre-RoPE query and key vectors in SDHs are nearly identical for all tokens, in contrast to more heterogeneous hidden states.

Implication: Because queries/keys are nearly invariant across positions, RoPE alone dictates the variation of the attention logits. The architecture thus enforces a decoupling of semantic content and positional structure in SDHs.

---

Fourier Analysis and Frequency Contributions

Given near-invariant queries/keys, the attention logits reduce to a structured sum over RoPE frequency components, mathematically analogous to a Fourier expansion:

$$\text{AttnLogit}(i, j) \approx \sum_\ell A_\ell \cos(\theta_\ell \cdot (i-j) + \varphi_\ell)$$

• Slash patterns as constructive interference: Peaks arise for particular values of $i-j=\Delta$, where the sum of cosine components is maximized due to constructive interference among dominant frequencies.
• Dominance of medium and high frequencies: Ablation experiments show removing high/medium frequencies severely degrades slash dominance, while low frequencies have negligible impact (Figure 4).

Figure 4: Removal of high- and medium-frequency RoPE components results in strong attenuation of slash-dominant attention scores.

---

Formal Sufficient Conditions via Training Dynamics

To theoretically ground the emergence of SDHs, the authors analyze the training dynamics of a two-layer attention-only transformer with RoPE, trained on an in-context learning regression task. They formalize two sufficient conditions for SDH emergence:

1. Token embeddings cluster on a cone: Enforces rank-one structure in queries/keys after projection.
2. RoPE frequency spectrum satisfies a slash-dominance condition: Constructive overlap occurs at specific offsets $\Delta$, recapitulating the empirical patterns.

Under these conditions, gradient descent provably drives the first-layer head to become an SDH (i.e., attention concentrated on sub-diagonal $\Delta$), and the property generalizes to OOD prompts and arbitrary $\Delta$. The theoretical results are consistent with empirical observations on training dynamics and relative phase transition times of different functional heads.

---

SDHs at Large Offsets

The framework extends to SDHs with very large $\Delta$ (e.g., $\Delta>500$). Despite much lower attention mass per entry, the same mechanisms (low-rank queries/keys, frequency interactions) hold, and OOD generalization persists (Figure 5).

---

Practical and Theoretical Consequences

Parameter Compression: Since queries/keys in SDHs are almost always rank-deficient, imposing rank constraints during training (or post-training compression) can yield considerable parameter efficiency without loss of accuracy.

Architectural Implications:

• The de-coupling of semantic content from positional signal in SDHs explains robust positional generalization, and suggests the non-semantic, architectural nature of many attention modules.
• The dominance of medium/high RoPE frequencies in SDH emergence informs future design and extension of the RoPE spectrum for long-context or specialized models.

Generalization to Other Position Embeddings: The mechanisms identified are specific to RoPE, but analogous phenomena may appear in other relative encoding schemes (e.g., ALiBi), warranting further research.

---

Conclusion

The study provides robust evidence—both empirical and theoretical—that SDHs and their characteristic slash patterns in RoPE-based LLMs are a consequence of intrinsic architectural interactions, primarily driven by nearly invariant pre-RoPE queries and keys, and constructive interferences among medium- and high-frequency RoPE components. This mechanistic understanding not only elucidates a central aspect of attention geometry in modern LLMs but also opens new directions toward interpretability, model compression, and principled architecture design.

---