Relay Autoregressive Fine-Tuning Strategy

• Relay Autoregressive (RAR) Fine-Tuning Strategy is a two-stage adaptation approach that decouples high-fidelity supervision from autoregressive decoding to handle long-range dependencies.
• It leverages iterative segmentation, external architectures, and strategic data augmentation to improve chain-of-thought reasoning and weather forecasting while reducing GPU memory usage.
• Empirical studies show that RAR recovers AR model accuracy from as low as 12% to 95% on long sequences and achieves efficient scaling in complex prediction tasks.

Relay Autoregressive (RAR) fine-tuning is a two-stage model adaptation strategy designed to overcome the limitations of classical autoregressive (AR) training when learning to generate or forecast long-range, structured outputs. RAR is motivated by the challenges of error accumulation, memory-compute inefficiency, and poor extrapolation in standard AR models. By leveraging external architectures or iterative segmentation to generate robust intermediate reasoning or state trajectories, RAR enables AR models to perform effectively on tasks with long sequences—such as chain-of-thought reasoning or medium-range weather prediction—without architectural modification or prohibitive cost. The approach has been instantiated in both sequence modeling domains ("Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning" (Yu et al., 12 Feb 2025)) and geophysical forecasting ("Searth Transformer" (Li et al., 14 Jan 2026)), and is characterized by strategic data augmentation, iterative supervision, and relay-based memory management.

1. Motivation and Theoretical Foundation

The primary challenge addressed by RAR fine-tuning lies in the breakdown of standard AR methods when forecasting or generating beyond the lengths seen during training. In AR Chain-of-Thought (CoT) models, as input length $n$ and problem complexity increase, the number of reasoning steps $m$ grows, often supra-linearly (e.g., polynomial in $n$) (Yu et al., 12 Feb 2025). This leads AR models with fixed positional encodings and length-limited data to collapse on long-range instances, with observed accuracy dropping from nearly $100\%$ in-distribution to below $20\%$ for extrapolation. In time series models such as weather forecasting, AR training over long horizons $T$ suffers from compounding errors and quadratic growth of memory usage in backpropagation through time (BPTT) (Li et al., 14 Jan 2026).

RAR fine-tuning addresses these deficiencies by decoupling the process of generating high-fidelity, long-range supervision (either through a model with superior length generalization, such as a Looped Transformer, or by segmenting the temporal horizon for memory efficiency), and then “relaying” explicit reasoning or trajectory data to fine-tune the AR model, thus preserving both the length-robustness of the external generator and the flexibility of AR decoding.

2. Formalism and Algorithmic Structure

RAR fine-tuning comprises two archetypal instantiations:

1. Loop-Aligned Reasoning for Sequence Models:
   • A Looped Transformer processes fixed-length input $x = [x_1,\dots,x_n]$ and iteratively refines a hidden state $e_t=f(e_{t-1})$ for $T$ steps, bypassing explicit token-wise unrolling.
   • Each loop iteration $t$ aligns with a reasoning “round” in the CoT chain $z_{1},…,z_{m}$, which is chunked so that the $t$-th segment is $\tilde z_t \in (V \cup \{\langle pad \rangle\})^n$.
   • Intermediate supervision is applied: at each iteration $t$, the model predicts $\tilde z_t$ from $e_t$, generating explicit CoT token segments, with a masked cross-entropy loss only on valid tokens and the final pad (Yu et al., 12 Feb 2025).
2. Segmented Relay for Long-Horizon Temporal Prediction:
   • The forecast horizon $T$ is partitioned into $M=T/k$ non-overlapping segments of length $k$.
   • Within each segment $s$, $k$ consecutive autoregressive predictions of the state (e.g., atmospheric grid $X_t$) are made.
   • The computational graph is detached at each relay point $\bar X_{s \cdot k}$ so that BPTT memory is bounded by $\mathcal{O}(k)$, not $T$, solving the scaling bottleneck (Li et al., 14 Jan 2026).
   • After each segment, the final output is used as the input state for the next, preventing gradient propagation beyond segment boundaries and suppressing error propagation.

3. Data Generation and Fine-Tuning Procedures

In the loop-aligned RAR framework for reasoning (Yu et al., 12 Feb 2025), after training the looped model with iteration-wise supervision, synthetic demonstration data is generated:

• For input lengths $n$ beyond seen training values, the looped model outputs a complete, step-aligned CoT trajectory $z$ and final answer $y$, forming extended-length triples $(x, z, y)$.
• This set $D_\text{ext}$ is merged with the original CoT data $D_\text{orig}$ over standard-length problems, yielding the augmented dataset $D_\text{aug} = D_\text{orig} \cup D_\text{ext}$.

The AR model is then fine-tuned on $D_\text{aug}$ with the objective: $$L_{\text{fine}} = \mathbb{E}_{(X,Z,Y)\sim D_\text{aug}} \left[ -\sum_{i=1}^m \log p_\text{AR}(z_i\mid z_{<i}, X) - \log p_\text{AR}(Y\mid z, X) \right]$$

In relay-based fine-tuning for temporal models (Li et al., 14 Jan 2026), the training proceeds as follows:

• Each relay segment predicts $k$ steps and accrues segment loss, e.g., for latitude-weighted MAE:

$$\ell(\hat X, X)=\frac{1}{C\cdot H\cdot W} \sum_{c,i,j} w_i |\hat X_{c,i,j}-X_{c,i,j}|, \quad w_i\propto\cos(\text{latitude}_i)$$

• The loss within a segment is backpropagated, and the final state is detached to seed the next segment.

4. Architectural and Implementation Characteristics

RAR fine-tuning is agnostic to model architecture and integrates with both sequence generators and temporal predictors.

• In reasoning chains, the Looped Transformer is equipped with an intermediate prediction head and trained jointly (iteration-wise and answer losses). No modifications to AR model architecture are required for fine-tuning.
• In Searth Transformer for weather prediction, the relay segmentation and detach operation are implemented at the data pipeline and training loop level. The core transformer structure—including windowed self-attention and skip connections—is maintained across both pre-training and RAR fine-tuning (Li et al., 14 Jan 2026).

Computation is optimized by mixed-precision arithmetic and gradient checkpointing (e.g., memory held below 25 GB on 8 A800 GPUs). The batch size, segment length $k$, learning rate (e.g., $1 \times 10^{-7}$ constant), and number of segments $M$ are configurable.

5. Complexity, Memory, and Optimization Analysis

RAR fine-tuning fundamentally alters the scaling profile of AR training:

Training Strategy	Memory Usage	Compute Cost	Horizon Capability
Classical AR	$\Theta(T)$	$\mathcal{O}(T)$	limited by hardware
RAR (segment length $k$)	$\mathcal{O}(k)$	$\mathcal{O}(T)$	scalable to large $T$

By detaching the computation graph at the end of each relay or loop-aligned segment, only activations for the current segment must be cached, enabling fine-tuning over $T\gg k$ steps without exceeding memory budgets (Li et al., 14 Jan 2026). This memory advantage translates directly to a substantial reduction in resource usage; for example, 15-day weather fine-tuning with RAR matches classical AR accuracy while reducing GPU-hour $\times$ GB by $\sim 200\times$.

6. Empirical Performance and Ablative Studies

RAR fine-tuning provides notable gains in both extrapolation and efficiency.

In loop-aligned CoT reasoning (Yu et al., 12 Feb 2025):

• AR-CoT accuracy on long sequences collapses from nearly $100\%$ in-distribution to $<20\%$ out-of-distribution, while the vanilla looped model maintains high performance over extrapolated lengths.
• After RAR-based fine-tuning with loop-generated chains, AR-CoT models regain almost the entire generalization gap, e.g., recovering from $12\%$ to $95\%$ accuracy on arithmetic problems of length 25.
• Hit-matrix analysis for LIS problems reveals $99\%$ CoT step accuracy with RELAY-generated chains, compared to $<60\%$ for AR self-generated chains after several steps.

In global weather forecasting (Li et al., 14 Jan 2026):

• On Z500 skill (ACC = 0.6), RAR fine-tuning achieves a skillful forecast of 10.3 days, surpassing both pre-trained baseline (8.7 days) and matching state-of-the-art at a fraction ($\sim1/200$) of the GPU resource.
• Variants adjusting $k$ and $M$ confirm that RAR accuracy scales smoothly with horizon, with virtually no degradation as $T$ increases.

Ablation studies emphasize the necessity of (a) iteration-wise intermediate supervision (for reasoning), and (b) explicit graph detachment at the segment boundary (for temporal models), as core contributors to RAR’s performance and generalization.

7. Significance, Interpretive Remarks, and Applicability

RAR fine-tuning provides a modular protocol that enables AR models to inherit the structural inductive biases or length-extrapolation capacities of supplementary architectures, while maintaining the operational and deployment advantages of AR decoding. The approach is applicable wherever long-range dependency, error propagation control, or resource-efficient fine-tuning is required—spanning reasoning engines, geophysical simulations, and other sequential prediction tasks.

Key takeaways include:

• Segmented, intermediate-supervised or relay-based training permits both better generalization and greater scalability versus classical AR or naive BPTT (backpropagation through time).
• RAR isolates the length generalization property in a separate model or training protocol and directly transmits this capability to AR models via staged data or segment-wise learning.
• Empirical results validate robust accuracy across a spectrum of task domains, attesting to RAR’s utility as a general mechanism for long-horizon autoregressive learning (Yu et al., 12 Feb 2025, Li et al., 14 Jan 2026).