Adapting language models to handle sequences longer than those seen during training

Develop effective methods to adapt pretrained transformer-based language models to handle inference sequences that exceed the training context length seen during pretraining.

Background

Transformer self-attention scales quadratically with sequence length, making pretraining at long contexts computationally prohibitive. When inference sequences exceed the training context, models that use explicit positional embeddings (such as RoPE) encounter out-of-distribution positional phases and sharply degraded performance.

Prior RoPE frequency-scaling methods (e.g., PI, RoPE-NTK, YaRN) aim to mitigate these issues but still typically require long-context finetuning and fail to generalize out-of-the-box on downstream tasks. This motivates the need for robust approaches to adapt LLMs to longer sequences beyond those seen during training.