Extreme-Token Phenomena in Deep Learning

• Extreme-token phenomena are training pathologies in Transformers where certain tokens attract excessive attention, leading to abnormal value-state drains and residual peaks.
• A dynamic mutual-reinforcement cycle between attention logits and value norms drives these effects, impairing model expressivity and stability.
• Mitigation strategies like Value-State Gated Attention break this cycle, enhancing model efficiency, quantization, and performance across tasks.

Extreme-token phenomena refer to a collection of training pathologies found in deep learning models that process tokenized data, especially Transformers, when faced with either problematic tokens (those acquiring outsized model attention or exhibiting abnormal internal representations) or with regimes of extreme token compression and segmentation. These phenomena encompass issues such as attention sinks, value-state drains, and residual-state peaks in neural architectures, as well as challenges in token reduction for efficient inference. The mechanisms underlying extreme-token behaviors critically affect the performance, interpretability, and operational efficiency of modern neural language, vision, and multimodal models.

1. Core Definitions and Manifestations

Within self-attention models, “extreme-token phenomena” consist primarily of three interrelated symptoms:

• Attention sinks: Specific tokens (often the beginning-of-sequence or nonsemantic placeholders) attract nearly all attention weight from multiple queries, irrespective of semantic relevance.
• Value-state drains: The value vectors $V_j$ of these sink tokens collapse toward zero norm as a result of the gradients, making them systematically appealing destinations for attention heads seeking a “no-op” output.
• Residual-state peaks: In deeper network layers, the residual representations of sink tokens exhibit abnormally large activations compared to other tokens.

These effects are unified by a mutual-reinforcement cycle: the softmax constraint enforces normalization across attention weights, so uninformative queries concentrate attention on certain tokens, which in turn get their value-states suppressed by the optimizer, making them even more attractive for future attention “dumping.” This cycle undermines model expressivity, complicates interpretability, and destabilizes quantization due to skewed activation distributions (Bu et al., 10 Oct 2025, Guo et al., 2024).

In morphologically rich languages, “extreme-token phenomena” also refer to situations where single orthographic tokens encapsulate multiple ambiguous word-units, requiring deep contextual understanding for segmentation (Brusilovsky et al., 2022).

2. Theoretical Analysis and Training Dynamics

The emergence of extreme-token phenomena is explained by a dynamical systems analysis of model optimization. In the case of the Bigram-Backcopy (BB) toy task, the training dynamics reveal a simple mutual-reinforcement mechanism between attention logits and value norms. Let $a_{i \to s}$ denote the attention from token $i$ to $s$. If, for most $i$, $a_{i \to s} \approx 1$, then the gradient with respect to $V_s$ becomes $$\partial L/\partial V_s \approx \partial L/\partial z_i$$, driving $\|V_s\| \to 0$. The smaller the value norm, the more reliably token $s$ serves as a gradient sink, feeding a positive feedback loop.

Analytical results on this mechanism demonstrate:

• The sink-logit concentration $\Delta\!\logit_{\cdot, s}$ grows logarithmically in step count, driving sharper attention condensation.
• The value vector of the sink token collapses, and residual activations for the sink climb linearly with training steps when using the Adam optimizer, leading to residual-state peaks.
• In real LLMs (e.g., Llama, OLMo), the same dynamics are observed, with heads entering “active” (contextual) or “dormant” (sink-forming) modes depending on input domain.

The formation of these pathologies is also shown to be sensitive to architectural choices (softmax vs. ReLU attention) and optimizer settings (Adam vs. SGD). Use of ReLU, for example, breaks the gradient-sharpening mechanism by the softmax, preventing value-state collapse. Substituting SGD for Adam eliminates the linear growth of residual peaks by allowing gradients to vanish as intended (Guo et al., 2024).

3. Architectural and Algorithmic Mitigations

Breaking the mutual-reinforcement cycle is critical for model stability and efficiency. The Value-State Gated Attention (VGA) mechanism addresses this by introducing a learnable, data-dependent gate $g_j = \sigma(V_j W_g)$ that directly modulates the contribution of token $j$'s value-state to the attention output:

$$z_i^{\mathrm{VGA}} = \sum_{j=1}^n \alpha_{ij} (g_j V_j)$$

This mechanism introduces an additional gradient path, allowing the model to sever the direct gradient to a sink token when its gate closes (i.e., $g_s \to 0$ as $\alpha_{is} \to 1$), thereby nullifying value-state drain and breaking the cycle at source (Bu et al., 10 Oct 2025).

Other prior approaches include:

Method	Principle	Limitation
Register Tokens	Prepends nonsemantic offload tokens	Static, context-independent, ↑kurtosis
Learnable Sink	Adds a fixed embedding for no-op attention	Non-adaptive, context-agnostic
Input-Gated (IGA)	Gates via functions of input embeddings	Fails to sever the direct gradient path
VGA (proposed)	Gates via emergent value-state	Fully decouples via reactive negative feedback

VGA achieves quantization-friendly activations, reduces maximum activation and kurtosis, and outperforms alternatives in terms of perplexity and post-quantization degradation (Bu et al., 10 Oct 2025).

4. Extreme Token Regimes in Compression and Scaling

In multimodal models, the “extreme-token” regime also refers to the operational point where the number of visual tokens is aggressively minimized for computational efficiency, often down to a single token per image or video (Li et al., 2024, Zhang et al., 21 Mar 2025). Scaling-law analyses reveal that, under fixed inference FLOPs, model error decreases much faster with LLM parameter count ($N$) than with token count ($T$), as quantified by:

$$\varepsilon(N, T) = A N^{-\alpha} + B T^{-\beta} + D$$

with $\alpha \gg \beta$ for visual reasoning. Hence, inference-optimal deployment uses the largest LLM affordable, with as few input tokens as possible (typically $T^*=1$ for $Q=0$ prompts). For text-centric tasks like OCR, the scaling law inverts ($\beta > \alpha$), and preserving more visual tokens is favorable (Li et al., 2024).

Algorithmically, this regime requires new token compression modules that maintain performance under severe token reductions. Query-based Convolutional Cross-Attention (QueCC) fuses the user query with CLIP-ViT tokens, applies regional local attention, and produces superior downstream accuracy at $T=1$ compared to prior token-merging schemes.

In video LLMs, Token Dynamics achieves $0.07\%$ of the original token count with less than $1.13\%$ performance drop, using an object-centric hash table, spatiotemporal key map, and cross-dynamics attention to preserve both static content and motion (Zhang et al., 21 Mar 2025).

5. Extreme Tokens in Tokenization and Morphologically Rich Languages

For languages with high token-internal complexity, extreme-token phenomena manifest as the need to segment single orthographic tokens into multiple meaning-bearing units, often with substantial morphological ambiguity. This is exemplified by Hebrew and Arabic, where a single token may correspond to several words, their reading determined only by broad sentential context.

The Char-to-char Attention Token Segmentation (CATS) model resolves such ambiguity by combining contextualized token embeddings (from models such as mBERT) with character-level sequence-to-sequence decoders. Empirical results show state-of-the-art segmentation F1 (95.84 for Hebrew, 98.57 for Arabic), as well as significant improvements in downstream POS tagging, dependency parsing, and NER relative to baselines. The segmentation-first pipeline outperforms joint segmentation-and-labeling objectives, as separating tasks allows more precise boundary decisions (Brusilovsky et al., 2022).

6. Sample Complexity and Convergence at Extreme Sequence Lengths

With the expansion of model context windows, understanding the convergence of attention as token count grows is crucial. The rate at which attention computed on $n$ tokens approaches its infinite-token (mean-field) limit, termed token-sample complexity, is characterized at two levels:

• Uniform convergence of the attention map occurs at rate $O(e^{8 R^2 H^2}/\sqrt{n})$ for queries of norm at most $R$ and attention horizon $H$.
• Moment convergence (for the distribution of attended tokens under a Lipschitz function $f$) is at rate $O(C'(R)/n^\beta)$ with $\beta = \frac{1}{2(1 + 32\Psi)} < \frac{1}{2}$ ($\Psi = \|\Sigma\|_2 H^2$).

In the hardmax regime ($\alpha \to \infty$), convergence slows further, with a rate $O(1/\sqrt{\ln n})$ for the mean. Empirical evaluations on synthetic data and BERT token embeddings confirm these theoretical rates, highlighting non-trivial sample complexity limitations as context size grows (Bohbot et al., 11 Dec 2025).

7. Implications, Best Practices, and Open Directions

Extreme-token phenomena have substantial ramifications for model design, training stability, interpretability, and deployment:

• Mitigation: Proactive mechanisms (VGA, ReLU attention, careful optimizer selection) should be integrated early in pretraining pipelines to prevent reinforcing pathologies.
• Compression & Efficiency: For compute-constrained inference, focus on sophisticated, query-aware token compression. Extreme reduction (down to a single token) is optimal for vision-LLMs where input token count is not a dominant factor in task performance.
• Segmentation Pipelines: In morphologically rich languages, segmentation-first approaches leveraging deep contextual cues deliver robust performance gains over joint models.
• Scaling Limitations: As models operate at ever-increasing sequence lengths, careful theoretical and experimental analysis of sample complexity is required to ensure reliability and stability.

Future research should target generalized, context-reactive gating mechanisms and further clarify the trade-off frontiers between token representation, compression, and model capacity across domains (Bu et al., 10 Oct 2025, Guo et al., 2024, Li et al., 2024, Zhang et al., 21 Mar 2025, Brusilovsky et al., 2022, Bohbot et al., 11 Dec 2025).