Attention Sink Tokens in Transformers

• Attention Sink Tokens are recurrent phenomena in transformers where specific tokens attract an unusually large share of attention, serving as geometric anchors.
• Their emergence results from softmax normalization and optimization dynamics, affecting representational geometry, inference, and memory through mechanisms like value drains and over-smoothing.
• Mitigation strategies, including alternative activation functions and dynamic token selection, efficiently redistribute attention to improve model streaming, quantization, and multimodal performance.

An attention sink token is a recurrent phenomenon in transformer-based architectures, characterized by the emergence of specific tokens—often, but not exclusively, initial or special tokens—that attract a disproportionately large share of the attention distribution from other sequence positions. While these tokens can be semantically uninformative, they profoundly shape model dynamics, impacting representational geometry, inference efficiency, memory and quantization strategies, and the overall stability of deep transformers. Recent research has rigorously elucidated the mathematical underpinnings, functional roles, and architectural determinants of attention sinks, as well as introduced a range of algorithms for both mitigating their negative effects and leveraging their properties for efficiency and interpretability.

1. Formal Definition and Mathematical Characterization

An attention sink token $s$ is formally defined, given an attention weight matrix $A^{(\ell)}\in\mathbb{R}^{N\times N}$ at layer $\ell$, as a token whose column sum $s^{(\ell)}_s=\sum_{i=1}^N A^{(\ell)}_{is}$ greatly exceeds that of other tokens, i.e., $s^{(\ell)}_s \gg s^{(\ell)}_j$ for all $j\neq s$ (Shin et al., 5 Jul 2025). The per-head version thresholds the mean incoming attention: token $j$ is a sink for head $h$ if $\frac{1}{N}\sum_{i=1}^N \alpha_h(i,j)>\tau_\alpha$ (Sandoval-Segura et al., 4 Apr 2025). In practice, the first token (e.g., BOS or [CLS]) is most commonly the sink position in autoregressive LLMs, but multiple or intermediate sinks can emerge, including those at punctuation or modality boundary tokens (Anand et al., 26 Oct 2025, Su et al., 6 Aug 2025).

Sinks also manifest in the geometry of hidden states: after early layers, the cosine similarity $\cos_\mathrm{sim}^{(\ell)}(h_i, h_s)$ between each token $i$ and the sink token $s$ increases monotonically, while $h_s^{(\ell)}$ itself remains nearly static across layers, making it a fixed attractor direction on the unit sphere (Shin et al., 5 Jul 2025, Anand et al., 26 Oct 2025). Tokens orthogonal to $h_s$ are those least aligned with the sink, typically corresponding to more informative representations.

2. Emergence Mechanisms and Theoretical Insights

The emergence of attention sinks is a consequence of the softmax normalization and transformer optimization, especially under over-parameterization and long context lengths. Softmax enforces a sum-to-one constraint, and when no token is semantically relevant, surplus attention is allocated to one or a few tokens that act as geometric "anchors," leading to spurious foci—attention sinks (Fu et al., 1 Jan 2026, Gu et al., 2024). This effect is exacerbated in deep and wide models, with the fraction of heads exhibiting sink behavior increasing with model depth and context length (Barbero et al., 3 Apr 2025).

Geometrically, attention sinks can be viewed as emergent reference points anchoring high-dimensional representation manifolds. Depending on architecture and positional encoding, reference frames may be centralized (single sink), distributed (multiple moderate sinks), or bidirectional (boundary tokens as anchors) (Ruscio et al., 4 Aug 2025). The formation of sinks is rapid and occurs early during pre-training, often within the first few thousand optimization steps, and becomes stable or slightly declines as training proceeds (Sandoval-Segura et al., 4 Apr 2025, Gu et al., 2024).

A mutual reinforcement dynamic connects attention sinks and "value drains": once a token's value norm $\|v_{j^*}\|$ diminishes, it draws disproportionately high attention, which in turn further suppresses its value norm, locking in the sink effect (Guo et al., 2024). This explains why sinks consistently have very low value norms and why their removal (e.g., via masking) leads to severe representational disruption.

3. Functional Roles in Sequential, Multimodal, and Diffusion Models

While often semantically empty, attention sink tokens play several structural and functional roles:

• Prevention of Representational Collapse: In long-context and deep transformers, concentrating attention on a sink decouples content tokens, preventing over-mixing and preserving information diversity (Barbero et al., 3 Apr 2025, Fu et al., 1 Jan 2026).
• Streaming and KV Cache Management: Off-the-shelf models rely on a small set of initial sinks to stabilize streaming inference. Retaining these tokens’ KV states is critical for infinite-context decoding; ablating them leads to catastrophic perplexity spikes (Xiao et al., 2023).
• Quantization and Outlier Features: Sink tokens' hidden state and KV statistics are extreme outliers (low norm but high cosine similarity), and their preservation is crucial for high-fidelity, low-bit quantization. Dynamic methods such as KVSink outperform static “preserve-first-N” strategies by accurately predicting which tokens act as sinks on each input (Su et al., 6 Aug 2025).
• Multimodal and Vision Models: Sink phenomena generalize to vision transformers and multimodal LLMs. Visual sink tokens are those with massive feature-norms or outlier activations; removing or reweighting attention away from them does not affect performance, permitting the redistribution of attention mass for improved visual grounding (Kang et al., 5 Mar 2025, Luo et al., 9 Oct 2025, Feng et al., 9 Apr 2025).
• Diffusion LLMs (DLMs): In DLMs, sinks are dynamic—appearing at different semantic or structural positions as denoising progresses. Unlike ARMs, masking these sinks has a minor impact, reflecting the model's robustness to anchor removal (Rulli et al., 17 Oct 2025).

4. Failure Modes, Pathologies, and Security Implications

Despite their utility, attention sinks are implicated in a range of undesirable effects:

• Redundancy and Dormant Heads: Many attention heads become dominated by sink focus ("dormant" heads), contributing little to downstream computation. Up to 20% of heads can be zeroed with minimal effect on accuracy, indicating substantial redundancy (Sandoval-Segura et al., 4 Apr 2025).
• Over-smoothing and Interference: Uniform high attention on sinks drives token representations toward the mean, causing feature collapse ("over-smoothing") and, in the continual learning setting, negative transfer and catastrophic forgetting through shared sinks (Bai et al., 2024).
• Backdoor Attacks and Unlearning: Sinks serve as “gateways” for the insertion of adversarial triggers or for restoring unlearned behaviors. Triggers placed at sink positions strongly amplify their impact, making unlearning processes vulnerable to backdooring (Shang et al., 19 Oct 2025).
• Compression and Model Pruning: The mechanism underpinning “catch, tag, and release” in few-shot learning depends critically on the low-rank parameter subspace generating both sinks and outlier features. Removing this component degrades both few-shot performance and streaming properties (Zhang et al., 2 Feb 2025).
• Massive Activations ("Dark Signals"): Sink tokens produce hidden states with anomalously large activations on a small subset of singular-vector directions ("U-dark" bands), a regime orthogonal to logit-sensitive subspaces. These activations are necessary for both low-loss inference and for absorbing unused attention mass (Cancedda, 2024, Anand et al., 26 Oct 2025).

5. Mitigation Strategies and Practical Algorithms

A substantial body of work has proposed algorithms to suppress, leverage, or redistribute attention sinks:

• Activation and Kernel Modifications: Replacing softmax with non-negativity-preserving alternatives (ReLU attention, Elastic-Softmax, sigmoid attention) eliminates the sum-to-one constraint, preventing automatic formation of sinks (Guo et al., 2024, Fu et al., 1 Jan 2026, Gu et al., 2024). Elastic-Softmax, in particular, adds a learned offset, permitting attention sparsity and eliminating spurious sink allocation while maintaining perplexity and task accuracy (Fu et al., 1 Jan 2026).
• Dynamic Token Selection (OrthoRank): By measuring tokens' orthogonality to the fixed sink direction, OrthoRank prioritizes the update of only the most informative, most orthogonal tokens, improving throughput and perplexity at matched sparsity (Shin et al., 5 Jul 2025).
• Streaming and Placeholding: Pre-training with dedicated learnable sink tokens allows a single anchor token to be used for streaming stabilization, outperforming both static and sliding-window baselines and requiring minimal architectural change (Xiao et al., 2023).
• KV Cache Quantization (KVSink): Outlier detection in intermediate layers accurately identifies all current sinks, enabling precise full-precision preservation and substantially reducing perplexity in low-bit inference (Su et al., 6 Aug 2025).
• Attention Redistribution (Multimodal/VAR/EAH): Methods such as Visual Attention Redistribution (VAR) and Enhancing Attention Heads (EAH) leverage sink identification to reallocate surplus attention toward informative visual tokens and mitigate object-level hallucination (Kang et al., 5 Mar 2025, Zhang et al., 2024).
• Regularization (Decorrelating Losses): Penalizing layer-wise cosine similarity between sink and non-sink tokens’ hidden states suppresses both attention and activation sinks, improving robustness to input compression in audio-visual and multimodal LLMs (Anand et al., 26 Oct 2025).
• Pre-scaling and Probing: Inserted scaling or probing layers, trained to rebalance class-token attention, increase deviation and reduce uniformity in sink attention, improving feature diversity and continual task performance (Bai et al., 2024).
• Task- or Behavior-specific Sinks: In structured domains such as click-through rate prediction, learned sink tokens are inserted between behaviors and trained to focus model attention at appropriate structural boundaries, enhancing prediction accuracy (Li et al., 5 Aug 2025).

6. Architectural Factors and Emergence in Practice

The formation and nature of attention sinks are determined by architectural choices, data distribution, and optimization:

Architectural Feature	Sink Pattern	Comments
Softmax normalization	Universal, strong sinks	Root cause; replaced by sigmoid eliminates sinks
RoPE positional encoding	Centralized (BOS) sinks	Cosine/geometric bias towards first token
NTK-aware/scaled RoPE	Distributed sinks	Weakens first-token bias
Learned/absolute positions	Bidirectional sinks	Alternating start/end token anchors
Data packing (always BOS 0)	Sink at fixed position	Removing sink collapses performance
Layer norm (pre- or post-)	Sink formation in both	With pre-norm, initial token norm "blows up"
Fine-tuning/LoRA	Sinks persist	Sinks robust to most downstream adaptation

Emergence is robust: in models from 14M to 70B parameters, with diverse encodings, sinks emerge rapidly after sufficient optimization and become a stable property of pre-trained weights (Gu et al., 2024, Barbero et al., 3 Apr 2025). The precise location and multiplicity of sinks can be modulated by data distribution (e.g., prefix randomization), mask layout (windowed vs. global), and pre-training corpus size.

7. Open Challenges and Future Directions

Despite the wealth of recent advances, attention sink research remains active across several fronts:

• Interpretability: Understanding the role of distributed and intermediate sinks beyond the canonical first-position anchor; relevance to model transparency and diagnosis.
• Security and Robustness: Mitigating the risk of backdoor triggers targeting sink tokens (Shang et al., 19 Oct 2025).
• Compression and Quantization: Formalizing the minimal set of weights and features needed to preserve sink-induced structure under aggressive compression (Zhang et al., 2 Feb 2025, Cancedda, 2024).
• Non-autoregressive and Diffusion Models: Characterizing how diverse architectural priors influence sink dynamics, especially for bidirectional or parallel refinement models (Rulli et al., 17 Oct 2025).
• Task-Conditioned Attention Allocation: Leveraging dynamic, context-aware sink selection (e.g., CTR-Sink, DIYSink) for structured prediction tasks (Li et al., 5 Aug 2025, Luo et al., 9 Oct 2025).
• Theory: Deepening the geometric and information-theoretic foundations connecting sink formation, reference-frame anchoring, and the tensor-geometry of attention (Ruscio et al., 4 Aug 2025).

Overall, the attention sink is a central phenomenon at the intersection of optimization, geometry, and architectural inductive bias in transformer models, with broad implications for future LLM design, deployment, and scientific understanding.