The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks

Abstract: We study two recurring phenomena in Transformer LLMs: massive activations, in which a small number of tokens exhibit extreme outliers in a few channels, and attention sinks, in which certain tokens attract disproportionate attention mass regardless of semantic relevance. Prior work observes that these phenomena frequently co-occur and often involve the same tokens, but their functional roles and causal relationship remain unclear. Through systematic experiments, we show that the co-occurrence is largely an architectural artifact of modern Transformer design, and that the two phenomena serve related but distinct functions. Massive activations operate globally: they induce near-constant hidden representations that persist across layers, effectively functioning as implicit parameters of the model. Attention sinks operate locally: they modulate attention outputs across heads and bias individual heads toward short-range dependencies. We identify the pre-norm configuration as the key choice that enables the co-occurrence, and show that ablating it causes the two phenomena to decouple.

• The paper demonstrates that step-up and step-down residual dynamics produce spike activations decoupled from necessary functional behavior.
• It reveals that normalization transforms massive activations into low-dimensional attention sinks, altering softmax routing and inference dynamics.
• Ablation studies show that head dimensionality and gating mechanisms modulate spike and sink intensity, guiding more efficient Transformer designs.

Disentangling Massive Activations and Attention Sinks in Transformer LLMs

Introduction and Problem Definition

This paper provides a comprehensive mechanistic analysis of two persistent phenomena in decoder-only, pre-norm Transformer LLMs: massive activations (extreme channel outliers in select tokens) and attention sinks (tokens that persistently attract a disproportionate share of attention mass across heads and layers). While both phenomena have critical implications for quantization, pruning, and inference efficiency, their functional relationship and causal origins have remained elusive. The authors rigorously demonstrate that their co-occurrence is not a necessary emergent property of Transformers, but rather an architectural artifact strongly influenced by normalization configuration and residual accumulation.

Emergence and Lifecycle of Massive Activations

Massive activations manifest as channel-wise spikes, restricted to a small set of intermediate layers and specific tokens. The authors trace their origin to distinctive step-up and step-down residual block dynamics in pre-norm architectures. Early in the network, certain feed-forward (FFN) blocks introduce massive, directionally-amplified outliers in a fixed set of channels, which are then propagated additively through subsequent layers by the residual stream. Near the network's end, other blocks (step-down) inject matching outliers of opposite sign, neutralizing the extreme values before output.

Figure 1: Top-3 channel magnitudes across depth in Llama 2 7B and Qwen3 8B, demonstrating abrupt rise, plateau, and neutralization of massive activations due to step-up and step-down blocks.

At the mechanistic level, these spikes arise from the FFN's quadratic form: the SwiGLU block operates in a near-identity regime for triggered tokens, and specific channel directions in weight space admit exceptionally high-gain quadratic amplification. The spike channels share highly aligned principal eigenvectors, so only inputs matching this direction (typically delimiter or first tokens) experience the amplification.

Figure 2: Input-output characteristics of SiLU in step-up/step-down blocks: spike tokens show nearly unchanged direction/norm, confirming SiLU's near-identity action.

Figure 3: Frobenius norms $\|U_k\|_F$ for quadratic forms: spike channels correspond to orders-of-magnitude outlier norm matrices, exclusively in step-up and step-down blocks.

Figure 4: Eigenvalue spectra for spike and non-spike channels: spike channels are sharply dominated by a principal eigenvalue, confirming rank-one directional amplification.

Mechanistic Account of Attention Sink Formation

Following the emergence of activation outliers, their high-magnitude values are mapped through RMSNorm in subsequent blocks. This normalization clamps their range, but in a high-magnitude regime, produces sparse, nearly token-invariant post-norm states. Thus, for any spike token, the normalized feature vector primarily occupies a fixed, low-dimensional channel subspace, with negligible variation across spike tokens.

Figure 5: Cosine similarity among spike tokens before and after step-up block: normalization collapses representations to nearly constant directions.

The key driver of sink phenomena then emerges: the geometric structure imposed by normalization ensures spike tokens (i.e., attention sinks) consistently project into unique, isolated regions of the attention head query/key space, enabling heads to erect large, persistent logit gaps favoring the sink token. When the per-head attention space is sufficiently wide (large $d_{\text{head}}$), this separation is robust and logit contrast is strong.

Figure 6: t-SNE visualization of query/key vectors for sink and non-sink heads: sink heads maintain close proximity between queries and the fixed spike key direction, whereas non-sink heads do not manifest such isolation.

Architectural and Training Factors: Ablation Studies

Targeted ablation experiments interrogate the causal levers underlying both spikes and sinks. Key findings include:

• Normalization Configuration: Adding post-block normalization (sandwich norm) or using elementwise dynamic transforms (DynamicTanh) completely suppresses massive activations, but has little effect on the prevalence of sinks. Conversely, QK-only normalization eliminates spikes but leaves sinks intact. This definitively demonstrates their decouplability.
• Feed-Forward Block Variants: Replacing SwiGLU with GeLU, a linear transform, or even attention-only blocks does not abolish sinks or spikes, but modulates their intensity, confirming that block-specific design is not causal, though it affects amplification efficiency.
• Head Dimensionality: Increasing $d_{\text{head}}$ sharply increases both sink ratio and spike magnitude. Distributing total capacity across more, smaller heads yields only marginal effects, confirming that per-head capacity (subspace dimensionality) is the geometric bottleneck for sink separation.
• Gated Attention and Training Distribution: Introducing dynamic, learnable gating (dependent on hidden state) abrogates the need for sinks — the attention routing is explicitly handled, so the learned 'sink' gating workaround disappears. Similarly, restricting the training context to only long sequences removes the need for sinks, confirming their role in short-range (local) dependency bias.

Empirical Regularities Across Model Families

The characteristic step-up/step-down activation profile and quadratic amplification are validated across all major open-source LLMs, including the Llama2/3 and Qwen2.5/3 families. Outlier Frobenius norms in quadratic forms, constrained to the same block locations and channel indices, also replicate universally.

Figure 7: Top-3 coordinate magnitudes before/after residual connections for 12 open models, all exhibiting the same step-up and step-down driven outlier lifecycle.

Figure 8: Frobenius norms of quadratic forms across Llama models; spike channels are uniquely associated with outlier norm values in specific blocks.

Theoretical Implications and Practical Consequences

The results robustly demonstrate that massive activations and attention sinks are not functionally necessary but structurally contingent artifacts of pre-norm designs. Their frequent co-occurrence is explained by the architectural link: unbounded residual accumulation allowed by pre-norm RMSNorm facilitates the formation of persistent, sparse almost-constant states at specific positions, upon which the softmax attention can easily construct stable sinks. However, either phenomenon can be eliminated independently (e.g., by changing norm placement or attention configuration) with negligible effect on perplexity — sink heads are replaced by alternative load-balancing or routing strategies when the architecture allows.

The theoretical insight is that attention sinks act as a learned, implicit gating mechanism, biasing heads toward short-range dependencies when explicit dynamic routing is unavailable. Their emergence is primarily due to (1) geometric isolation of normalized spike token representations and (2) the preference for local recency in the pretraining context-length distribution.

On the practical front, these findings furnish direct pathways for efficiently mitigating quantization and inference challenges (such as activation outlier suppression and cache management) without compromising core modeling capacity. Furthermore, refining normalization layers and attention routing may yield more robust, interpretable, and hardware-friendly Transformer variants.

In future directions, these results suggest that architectural design targeting precise routing and representation compression (e.g., dynamic gating or normalization-free blocks) can eliminate the training-driven incentive for sink formation, supporting more scalable and efficient LLMs.

Conclusion

This work provides a precise, mechanistic foundation for understanding the intertwined but ultimately independent natures of massive activations and attention sinks in LLMs (2603.05498). The analysis conclusively demonstrates that both phenomena are architectural byproducts rather than functional prerequisites and sets the stage for targeted model design and optimization that address their respective drawbacks without incurring performance penalties. The insights herein are expected to influence both theory and practice regarding normalization, residual design, and attention routing for large-scale Transformers.