Sliding Window Attention Training for Efficient Large Language Models

Abstract: Recent advances in transformer-based LLMs have demonstrated remarkable capabilities across various tasks. However, their quadratic computational complexity concerning sequence length remains a significant bottleneck for processing long documents. As a result, many efforts like sparse attention and state space models have been proposed to improve the efficiency of LLMs over long sequences. Though effective, these approaches compromise the performance or introduce structural complexity. This calls for a simple yet efficient model that preserves the fundamental Transformer architecture. To this end, we introduce SWAT, which enables efficient long-context handling via Sliding Window Attention Training. This paper first attributes the inefficiency of Transformers to the attention sink phenomenon resulting from the high variance of softmax operation. Then, we replace softmax with the sigmoid function and utilize a balanced ALiBi and Rotary Position Embedding for efficient information compression and retention. Experiments demonstrate that SWAT achieves SOTA performance compared with state-of-the-art linear recurrent architectures on eight benchmarks. Code is available at https://github.com/Fzkuji/swat-attention.

• The paper introduces Sliding Window Attention Training (SWAT) to reduce quadratic complexity to linear for processing long sequences in Transformers.
• It replaces softmax with sigmoid and integrates a sliding window mechanism with balanced ALiBi and RoPE to ensure stable training and improved efficiency.
• Experimental results demonstrate state-of-the-art performance on commonsense reasoning benchmarks, showcasing superior long-context handling.

Sliding Window Attention Training for Efficient LLMs

Introduction

The paper "Sliding Window Attention Training for Efficient LLMs" (2502.18845) aims to address the computational bottleneck associated with processing long sequences in transformer-based LLMs. Traditional LLMs utilize architectures that suffer from quadratic complexity with respect to sequence length, necessitating more efficient methods to handle long-context tasks. This paper introduces Sliding Window Attention Training (SWAT), a novel approach that maintains the simplicity of the fundamental Transformer architecture, enabling efficient long-context handling without the need for complex architectures or parallel training techniques.

Sliding Window Attention Mechanism

SWAT proposes the replacement of softmax with the sigmoid function to achieve efficient information compression and retention. This substitution is part of a sliding window attention mechanism, which limits each token's attention calculation to its neighboring tokens within a fixed-size window. This modification reduces computational complexity from quadratic to linear, allowing SWAT to maintain model performance while processing long sequences.

Figure 1: The demonstration of the SWA mechanism in Transformers.

In SWAT, the sigmoid function prevents attention sink phenomena, a condition where models allocate excessive attention to initial tokens, causing uneven distribution across the sequence. Unlike softmax, sigmoid activation maintains dense attention weights, allowing for higher information capacity per token.

Model Architecture and Efficiency

SWAT incorporates balanced ALiBi and Rotary Position Embedding (RoPE) to stabilize the training process. Balanced ALiBi introduces position-dependent differentiation across bidirectional attention heads, enabling effective information preservation from recent and historical contexts. The incorporation of RoPE adds explicit positional information to hidden states, ensuring stable training.

Figure 2: The demonstration of the SWA mechanism in Transformers, where the model's information coverage includes residual and active tokens, depending on the model depth and window size.

This approach ensures that the architecture closely resembles standard attention layers, with negligible additional overhead from ALiBi computation. Furthermore, SWAT achieves linear complexity during inference with sliding window attention, significantly improving efficiency while maintaining performance across tasks.

Experimental Results

SWAT demonstrates state-of-the-art performance across eight commonsense reasoning benchmarks, achieving superior results with linear recurrent architectures. The experimental results validate SWAT's efficacy, showing substantial improvements in both language modeling and reasoning tasks when compared to traditional models.

Figure 3: The training loss of models with different modules including Sigmoid, RoPE, and ALiBi, with the balanced slopes.

SWAT's ability to handle long-context dependencies and bridge the training-inference gap illustrates its potential to outperform existing approaches relying on complex architectures while preserving the convenience of implementation.

Discussion and Future Directions

The implications of this research suggest that sliding window attention significantly enhances long-sequence processing in LLMs, enabling efficient handling of complex reasoning tasks. Practical deployment of SWAT in environments constrained by computational resources could lead to more scalable and adaptable AI applications.

Future research may explore adaptive window sizes or hybrid configurations to further optimize processing efficiency. Additionally, extending SWAT to accommodate ultra-long sequences could involve integrating explicit memory retrieval mechanisms to ensure complete information retention over extensive contexts.

Conclusion

The introduction of SWAT provides a promising avenue for efficient LLMs by leveraging sliding window attention training within existing Transformer architectures. By replacing softmax with sigmoid and incorporating balanced ALiBi and RoPE, SWAT effectively addresses critical challenges in long-context processing while achieving state-of-the-art performance.