Exclusive Self Attention

Abstract: We introduce exclusive self attention (XSA), a simple modification of self attention (SA) that improves Transformer's sequence modeling performance. The key idea is to constrain attention to capture only information orthogonal to the token's own value vector (thus excluding information of self position), encouraging better context modeling. Evaluated on the standard language modeling task, XSA consistently outperforms SA across model sizes up to 2.7B parameters and shows increasingly larger gains as sequence length grows.

• The paper introduces Exclusive Self Attention that orthogonalizes self-value output to reduce attention bias.
• It employs a parameter-free operation with negligible compute overhead, improving training convergence and generalization.
• Empirical tests show superior performance on language modeling and long-context tasks across various Transformer scales.

Exclusive Self Attention: Orthogonalization for Enhanced Context Modeling in Transformers

Overview

This paper introduces Exclusive Self Attention (XSA), a simple yet effective modification to standard self-attention (SA) mechanisms in Transformers. The central innovation is the explicit exclusion of self-value vector information from the attention output, enforcing a strict decomposition of point-wise feature transformation (delegated to FFN) and context aggregation (handled by SA). This approach is empirically shown to mitigate the "attention similarity bias," improve language modeling performance across model scales and sequence lengths, and maintain computational efficiency. The evaluation is conducted on the FineWeb-100BT corpus, with additional downstream task assessment.

Motivation and Problem Analysis

The standard SA mechanism aggregates information from both self and contextual tokens, with the FFN providing position-wise transformation. However, the authors demonstrate a strong "attention similarity bias": the output of the SA layer exhibits high cosine similarity with the corresponding self-value vector $v_i$. This behavior amplifies with network depth and implies redundancy, as the FFN already processes the point-wise transformation via the residual path. This redundancy leads to suboptimal context utilization (Figure 1).

Figure 1: The attention output in standard SA is highly similar to the self-value vector, indicating attention similarity bias that grows deeper in the stack.

Empirical analysis of a 1.3B parameter model reveals two salient phenomena:

• Value vectors $v_i$ and $v_j$ are positively correlated within a sequence.
• Diagonal attention scores $a_{i,i}$ are consistently high.

Together, these imply SA overemphasizes the self-point, compromising context modeling—a critical capability for sequence modeling and generalization.

Exclusive Self Attention Mechanism

XSA introduces an orthogonalization step at the attention output stage:

$z_i = y_i - (y_i^T v_i) \frac{v_i}{\|v_i\|_2^2}$

where $y_i$ is the standard SA output and $v_i$ the self value. This step projects out the component of $y_i$ aligned with $v_i$, ensuring that the output $z_i$ is orthogonal to the self information. As such, XSA decouples context modeling from point-wise transformation, which is handled by the residual connection to the FFN.

This operation is parameter-free, amenable to two lines of code (shown in PyTorch-style pseudocode in the paper), and does not compromise the expressive power of the overall architecture, as the complete information flow is preserved via the Transformer block's residual path.

Computational Efficiency

A baseline concern for augmentations to SA is their impact on speed and memory efficiency. Experiments on various sequence lengths and model sizes show that incorporating XSA introduces negligible computational overhead relative to vanilla SA (Figure 2).

Figure 2: XSA incurs minimal speed and memory overhead across sequence lengths and model scales.

Empirical Evaluation

Training Loss and Generalization

XSA is compared to baseline Transformers with model sizes of 0.7B, 1.4B, and 2.7B parameters, all trained on the FineWeb-100BT dataset:

• Training and validation loss: Across all scales, XSA achieves consistently lower training and validation loss than the baseline throughout training (Figure 3).

Figure 3: XSA delivers superior convergence and validation loss compared to baseline Transformer for all model sizes.

• Downstream tasks: Evaluated on eight language understanding and reasoning benchmarks, XSA models consistently surpass their SA counterparts in average accuracy, with the performance margin increasing with model size. These improvements are observed not only in aggregate, but also in individual tasks, though some cases show reduced performance in specific tasks (e.g., OBQA at 0.7B and 1.3B).

Robustness to Optimization Hyperparameters

XSA exhibits robustness to optimizer settings, maintaining performance superiority across a sweep of learning rates. The empirical loss margin is stable, confirming that the observed benefits are not hyperparameter-specific (Figure 4).

Figure 4: The reduction in training and validation loss for XSA is maintained across diverse learning rates.

Sequence Length Scaling

A notable finding is the increased effectiveness of XSA as sequence length grows. For longer contexts (up to 16,384 tokens), XSA's advantage over standard SA becomes more pronounced (Figure 5). This result is significant given the increasing demand for long-context modeling in modern NLP applications.

Figure 5: XSA's relative improvement over baseline grows with longer sequence lengths, highlighting its utility in long-context modeling.

Interaction with Attention Sinks

Unlike explicit Attention Sinks (learned tokens prepended as context pools), XSA implicitly manages undesired attention by zeroing out the self-component. Comparative analysis shows that XSA retains its performance margin even when explicit attention sinks are present (Figure 6).

Figure 6: The advantage of XSA persists with varying numbers of attention sinks.

Theoretical and Practical Implications

The explicit orthogonalization enforced by XSA operationalizes a more principled division of labor within the Transformer block. This modification directs attention capacity exclusively to contextual modeling, circumventing a common redundancy in existing architectures. The minimal computational penalty and robustness to hyperparameters suggest that XSA can be widely and seamlessly integrated into current Transformer-based pipelines.

Practically, the positive scaling in both model size and sequence length make XSA a favorable candidate for large-scale LLM pretraining and downstream applications requiring long-context synthesis. The consistency of performance improvement across tasks also points to generality in the observed effect.

Theoretically, these results motivate further analysis into the internal representations of self-attention layers and open the question of whether additional orthogonalization or decomposition strategies (possibly dynamically constructed) can further improve model efficiency and representation disentanglement.

Conclusion

Exclusive Self Attention introduces a straightforward yet powerful architectural enhancement to Transformers by orthogonalizing the attention output relative to the self-value vector. This intervention effectively resolves attention similarity bias, leading to clear improvements in language modeling loss and diverse downstream tasks. XSA's scaling properties, computational cheapness, and robust empirical performance indicate significant potential for adoption in both research and large-scale production settings. Future work should examine its efficacy at even larger scales, its compatibility with newer optimizers and architectures, and its transferability to other domains beyond language modeling.