UltraLong-8B Transformer Models

• UltraLong-8B models are 8-billion parameter transformers extended to support ultra-long context windows (up to 4M tokens) through continued pretraining and specialized data composition.
• They employ innovative YaRN rotary positional embedding scaling, allowing efficient attention over extremely long sequences while preserving performance on standard benchmarks.
• The architecture leverages context and tensor parallelism to manage quadratic compute costs, enabling robust retrieval and reasoning across very long inputs.

UltraLong-8B models refer to a family of 8-billion-parameter transformer-based LLMs that have been extended to operate with ultra-long context windows, from 128K tokens up to 4 million tokens, via efficient continued pretraining strategies and associated engineering techniques. Building on architectures such as Llama-3.1-8B, these models use scaling methodologies—particularly the YaRN rotary positional embedding recipe—to facilitate efficient attention-based processing over unprecedented sequence lengths. UltraLong-8B models achieve state-of-the-art performance on both long-context and standard general-purpose benchmarks, demonstrating practical feasibility and robustness for tasks requiring persistence or retrieval across extremely long input sequences (Xu et al., 8 Apr 2025).

1. Architecture Extension and Pretraining Pipeline

UltraLong-8B models begin with a base transformer—e.g., Llama-3.1-8B-Instruct (originally supporting 128K-token contexts)—and extend the context window to 1M, 2M, or 4M tokens in a single step via continual pretraining. This is accomplished with minimal architectural or hyperparameter modifications. The main technical elements are as follows:

• Document Concatenation with Special Separators: Documents are concatenated end-to-end with a single separator token (“<s>”), and no cross-document masking is employed. This enables the model to learn dependencies across the entire pretraining sequence length ($L_t$), rather than being constrained by artificial segment or document boundaries.
• Data Composition: The pretraining corpus consists of upsampled long documents (>$8$K tokens) and downsampled short documents (<$4$K tokens), focusing model capacity on the modeling of long-range dependencies and discouraging overfitting to trivial short-range contexts.
• Continual Pretraining Hyperparameters: A single pass of 1 billion tokens (one epoch) is used, with Adam optimizer ($\beta_1=0.9$, $\beta_2=0.95$, learning rate $3\times10^{-5}$), and trained on 256 NVIDIA H100 GPUs. Context parallelism ($cp$) is scaled (e.g., $cp=4$ for 1M, $cp=16$ for 4M) to distribute the quadratic attention cost.
• Rotary Positional Embedding (RoPE) Scaling: The YaRN (α=1, β=4) scaling recipe is employed to rescale angular frequencies, so that distinguishing power is preserved up to the full window $L_t$. For UltraLong-8B, scaling factors $s=\{128, 256, 512\}$ handle $L_t=\{1\,\text{M}, 2\,\text{M}, 4\,\text{M}\}$, respectively.

2. Rotary Positional Embedding Scaling Methodology

Standard RoPE assigns base frequencies for each attention head dimension $k$ as

$\omega_k = 10000^{-2k/d}$

with $d$ the head dimension. To accommodate ultra-long contexts, YaRN modifies the rotary phase for each position $pos$:

$\phi_k(pos) = s \cdot pos \cdot \omega_k$

where $s$ is chosen based on the maximum desired context length. Unlike alternatives such as NTK-aware scaling (which collapses out-of-distribution at extreme sequence lengths), YaRN’s polynomial scaling maintains robust performance across the entire window (Xu et al., 8 Apr 2025). Ablation indicates that NTK-scaling performs slightly better at short range, but YaRN avoids phase collisions that degrade accuracy beyond 128K tokens.

3. Instruction Tuning for Ultra-Long Contexts

Fine-tuning leverages a supervised blend of 100K short-context SFT examples (<8K tokens), notably from the AceMath-Instruct blend, which aggregates general, mathematical, and coding datasets (e.g., ShareGPT, OrcaMathWordProblems, Magicoder, GlaiveCodeAssistant). Responses are generated and refined by frontier models (e.g., GPT-4o). No synthetic long-context instructions are required; tuning only on short data preserves both reasoning and instruction-following abilities. Fine-tuning employs Adam with $5\times10^{-6}$ learning rate, and runs efficiently (~30 min/model) due to parallelization.

4. Empirical Performance and Benchmarking

UltraLong-8B models achieve state-of-the-art results on a range of long-context and standard tasks without significant regression on short-context benchmarks:

Model	RULER <128K	RULER <1M	InfiniteBench	MMLU	MATH	HumanEval
Llama-3.1 (128K)	88.3	61.3	24.7	64.83	47.22	69.51
ProLong-512K	82.4	77.8	28.6	48.33	15.12	35.97
UltraLong-8B-1M	86.6	79.1	32.1	66.99	55.10	68.29
UltraLong-8B-2M	85.0	78.2	32.5	67.31	51.36	67.07
UltraLong-8B-4M	84.2	78.0	30.4	65.14	50.92	67.68

• Needle-in-a-Haystack (NIAH) Retrieval: UltraLong-8B achieves 100% retrieval accuracy across 1M, 2M, and 4M windows, whereas both Llama-3.1-8B and ProLong-512k degrade substantially past 128K.
• Standard QA and Reasoning: UltraLong-8B matches or outperforms the 128K baseline on MMLU, MATH, and HumanEval, while ProLong and Gradient models exhibit substantial drop-offs for broader context extension.
• Ablations: Special document separators (“<s>”) enhance performance in multi-document retrieval (RULER, LV-Eval, InfiniteBench). Direct single-step extension from 128K to 1M+ tokens outperforms staged (multi-step) context length extension.
• Short-Context Calibration: There is negligible to marginal (≈1 point) reduction in average pass rates on short-context benchmarks after ultra-long-context training.

5. Computational Complexity and Engineering Considerations

The O($L^2$) compute and memory requirements inherent to full attention present practical barriers as $L$ grows. UltraLong-8B mitigates this via:

• Context Parallelism: Splits tokens across multiple GPUs, distributing both storage and computational load; $cp=4$ suffices up to 1M, $cp=16$ up to 4M tokens (using 256 NVIDIA H100 GPUs).
• Tensor Parallelism: Splits hidden dimension $d$ across GPUs ($tp=8$), further distributing projected parameters and reducing per-device burden.
• Empirical Resource Use: 1M-token context models train in ≈5 h, 4M-token ones in ≈13 h, reflecting the quadratic scaling but making training tractable for academic or enterprise users with high-performance compute clusters.

Inference latency also scales quadratically with $L$, but can be amortized across multiple simultaneous requests or partially mitigated by using efficient attention kernels (e.g., FlashAttention). No architectural modifications to attention sparsity or memory compression are included in the base recipe.

6. Comparison with Related Long-Context Model Strategies

Granite-8B (Stallone et al., 2024) previously extended code LLMs to 128K tokens via (1) progressive RoPE frequency scaling, (2) repository-level file packing, and (3) length-upsampling. Granite’s multi-stage gradual approach differs from UltraLong-8B’s single-step YaRN scaling and corpus up/downsampling. Both approaches demonstrate that—given modest continued pretraining (<0.1–1% of total tokens)—transformers with standard attention and RoPE can generalize to much longer contexts, provided positional encoding scaling and data composition are managed.

UltraLong-8B, however, systematically benchmarks direct scaling to multi-million token windows, establishes clear ablation on separator tokens and scaling strategy, and maintains or improves short-context performance even at the 4M-token scale. Models like ProLong-512K and Gradient-1048K show that alternatives can degrade either long- or short-context accuracy, especially with less robust frequency scaling or less careful corpus construction.

7. Applications and Limitations

UltraLong-8B is practical for tasks such as document/video understanding, retrieval-augmented generation, and scientific/technical applications requiring reasoning or search across very long inputs. The primary limitation remains the quadratic time/memory cost of full attention, although context parallelism makes training and inference at multi-million-token scales tractable with modern multi-GPU infrastructure.

There is no explicit ablation on data upsampling vs. RoPE scaling for UltraLong-8B; however, evaluation suggests that the majority of gains derive from proper RoPE scaling, with corpus engineering further ensuring robust performance.

UltraLong-8B models are made publicly available, facilitating reproducibility and further research into ultra-long context scaling (Xu et al., 8 Apr 2025).