Autoregressive LLMs: Architecture & Advances

• Autoregressive LLMs are transformer-based sequence models that predict tokens left-to-right using next-token conditional factorization.
• They employ masked self-attention and KV caching techniques to mitigate sequential dependencies and computational bottlenecks.
• Innovations like APAR, linear attention, and hybrid AR/infilling yield enhanced throughput, reduced latency, and cross-modal applications.

Autoregressive LLMs (AR-LLMs) are a class of transformer-based sequence models that define the joint probability of an output sequence through a left-to-right, next-token conditional factorization. Their adoption has shaped the landscape of natural language processing, generative modeling, and foundation model development. This article provides a comprehensive technical overview of AR-LLMs: their computational structure, training and inference paradigms, known limitations, extensions and hybridizations, efficiency bottlenecks and remedies, and theoretical universality.

1. Mathematical Formulation and Computational Structure

AR-LLMs model a discrete sequence $x = (x_1, \dots, x_T)$ over a fixed vocabulary $\mathcal{V}$ by the joint probability

$$p_\theta(x_1, \dots, x_T) = \prod_{t=1}^T p_\theta(x_t | x_{<t}),$$

where $p_\theta(x_t | x_{<t})$ is computed by a deep stack of masked self-attention and feed-forward sublayers, i.e., a transformer architecture. For each timestep $t$, the model computes hidden states $s_t = f_{\theta,t}(x_{t-\Delta+1:t-1})$ using the most recent $\Delta$-length causal context. The output layer projects hidden states to logits over the vocabulary via a parameterized affine transformation and softmax: $$p_\theta(x_t | x_{<t}) = \operatorname{softmax}(W_\text{out} s_t + b_\text{out}),$$ where $W_\text{out} \in \mathbb{R}^{V \times d}$ and $d$ is the model dimension. Training proceeds by minimizing the negative log-likelihood (cross-entropy) summed over all tokens in large corpora (Krishnamurthy, 31 Jan 2026, Pan et al., 10 Oct 2025).

Self-attention within AR-LLMs follows the paradigm of bilinear–softmax–linear operations for each head and each layer: $$q_{l,t}^k = W_Q^{k,l} h_{l-1,t},\quad k_{l,s}^k = W_K^{k,l} h_{l-1,s},\quad v_{l,s}^k = W_V^{k,l} h_{l-1,s}$$

$$\alpha_{l}^{k}(t,s) = \frac{\langle q_{l,t}^k, k_{l,s}^k \rangle}{\sqrt{d_h}},\quad w_{l}^k(t, s) = \operatorname{softmax}_s(\alpha_{l}^k(t,s))$$

$$\operatorname{Attn}_l^k(t) = \sum_{s \in C_t} w_{l}^k(t, s) v_{l,s}^k,$$

with $C_t = \{t-\Delta+1, \dots, t\}$ (causal mask), $H$ heads and $L$ stacked layers (Krishnamurthy, 31 Jan 2026).

2. Inference Mode, Decoding Algorithms, and Efficiency Bottlenecks

At inference, AR-LLMs emit output tokens sequentially: at step $t$, the next token $x_t$ is sampled or beam-searched from $p_\theta(x_t|x_{<t})$, appended to the context, and the process iterates. Efficient implementation requires caching key and value activations (KV cache) for each token-step, reducing per-token compute to $O(\Delta d^2)$. However, the strictly sequential dependency constrains wall-clock decoding speed, especially for long sequences (Liu et al., 2024, You et al., 2024):

• Sequential Dependency: No future token $x_t$ can be generated until all prior $x_{<t}$ are finalized. Latency and throughput scale linearly with $T$.
• Attention Complexity: Standard softmax attention incurs $O(n^2)$ compute and memory with sequence length $n$, limiting context length and real-time feasibility.
• KV Cache Growth: KV memory footprint increases with sequence length, limiting concurrency and maximum batch size.

These bottlenecks have prompted the development of both algorithmic and architectural innovations, as well as new hybridization schemes.

3. Training Paradigms, Alignment, and Robustness

The primary training objective is left-to-right next-token prediction, i.e., optimizing cross-entropy loss: $$\mathcal{L}_\mathrm{AR}(\theta) = -\mathbb{E}_{x \sim \mathcal{D}} \sum_{t=1}^T \log p_\theta(x_t|x_{<t})$$ Fine-tuning for alignment or downstream adaptation incorporates human preference data or specialized tasks. All standard alignment methods—Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), Rejection Sampling Fine-Tuning (RSFT), Reinforcement Learning from Verifiable Rewards (RLVR)—are instances of KL-regularized policy optimization: $$\max_\pi \mathbb{E}_{x, y \sim \pi} [R(x, y) - \beta\,\mathrm{KL}(\pi(\cdot|x)\,\|\ \pi_\text{ref}(\cdot|x))]$$ DPO, for example, provides a tractable supervised loss for pairwise preference data (Krishnamurthy, 31 Jan 2026, Lin et al., 2 Feb 2026).

Empirically, AR-LLMs exhibit the "reversal curse"—they struggle to answer questions that reverse information order seen at training. This is a consequence of the model’s causal factorization, which prohibits gradients at token $x_i$ from influencing predictions at positions $<i$ (Pan et al., 10 Oct 2025). Masked diffusion LLMs (dLLMs) and masked fine-tuning paradigms for AR-LLMs can mitigate these limitations, enhancing knowledge injection and bidirectional reasoning.

4. Efficiency Enhancements: Parallelism and Attention Linearization

Addressing the sequential and quadratic bottlenecks, several techniques have emerged:

Auto-Parallel AR Decoding (APAR)

APAR introduces hierarchy-aware parallelism via a “paragraph tree” structure. Fine-tuning on hierarchical training data enables the use of two special tokens, [Fork] and [Child], to spawn and disambiguate child decoding threads. The attention mask is modified such that a token attends to its prefix and ancestors in the hierarchy—reducing attention span up to 35%. APAR achieves a 2× speedup over baseline AR decoding, and up to 4× with speculative decoding. Empirical measurements confirm 20–70% higher throughput and 20–35% reduced latency in high-concurrency scenarios, with ≤2% degradation in output quality (Liu et al., 2024).

Linear Attention and Speculative-Compatibility

Linear attention mechanisms approximate the O($n^2$) softmax kernel using random features, low-rank projections, or grouped summations, often formalized as

$$\mathrm{Softmax}(QK^T)V \approx \phi(Q)(\phi(K)^TV)$$

Augmentations for AR-LLMs combine causal, masked depth-wise convolutions and grouped processing to recover parallelism and reinforce local dependencies while preserving strict causality. These augmentations result in up to 6.67× lower perplexity and 2× higher throughput on representative LLMs (You et al., 2024).

Hybrid AR/Infilling and Block Diffusion

To combine the benefits of bidirectional context and AR efficiency, MARIA fuses AR and MLM representations using a learned linear decoder: $$h_\mathrm{concat}^{(i)} = [h_\mathrm{AR}^{(i)};\ h_\mathrm{MLM}^{(i)}] \quad \rightarrow\quad \ell^{(i)} = W_3 h_\mathrm{concat}^{(i)} + b_3$$ MARIA delivers masked infilling with AR-level throughput, achieving state-of-the-art perplexity and sample quality relative to both MLM and diffusion baselines (Israel et al., 9 Feb 2025).

Fast-dLLM v2 converts a pretrained AR-LLM into a block diffusion LLM. By fine-tuning with complementary masking and hierarchical caching, it enables blockwise parallel decoding, matching or surpassing AR generation quality while delivering up to 2.5× inference speedup. Its hierarchical cache design supports intra-block bidirectional refinement and efficient context reuse (Wu et al., 30 Sep 2025).

5. Extensions and Cross-Modal Transfers

AR-LLMs have been successfully extended to non-text domains and hybrid multimodal models:

• Autoregressive Representation Alignment (ARRA) augments the standard next-token objective with a global visual alignment loss on a hybrid token. By distilling semantic representations from fixed visual encoders (e.g., CLIP), ARRA enables globally coherent and spatially consistent text-to-image generation while retaining standard AR inference. FID reductions of 4–25.5% on major image datasets confirm the effectiveness of plug-and-play AR-based multimodal modeling (Xie et al., 10 Mar 2025).
• AR-MAP weight transfer: Well-aligned AR-LLMs can serve as implicit teachers for diffusion LLMs by transferring the “preference delta” via simple weight scaling. This process preserves model architecture, sidesteps high-variance ELBO estimation in diffusion models, and achieves competitive or superior alignment on math, helpfulness, and truthfulness tasks (average score: 69.08%) (Lin et al., 2 Feb 2026).
• Production and Continual Learning: Techniques such as mask-reconstruction fine-tuning close the data-efficiency gap between AR-LLMs and dLLMs, improving ability to handle arbitrarily ordered knowledge and enabling incremental adaptation without catastrophic forgetting (Pan et al., 10 Oct 2025, Krishnamurthy, 31 Jan 2026).

6. Theoretical Universality and Computational Expressiveness

AR-LLMs with standard next-token decoding can be shown to be computationally universal. When equipped with a generalized sliding-window mechanism (bounded context and extended context concatenation), AR decoding simulates universal “Lag systems”—a form of production-rule computation equivalent to universal Turing machines. In a demonstration, 2027 explicit production rules were encoded as prompt instructions for gemini-1.5-pro-001, confirming deterministic, Turing-complete computation via greedy decoding alone. This result establishes AR-LLMs as general-purpose computers by the Church–Turing thesis, with universality emerging from the architecture and protocol rather than from explicit training interventions (Schuurmans et al., 2024).

7. Limitations, Trade-offs, and Open Directions

AR-LLMs, while expressive and robust under left-to-right generation, exhibit several empirical and theoretical limitations:

• Reversal curse and data efficiency: Causal factorization impedes reverse-style QA and knowledge retrieval unless mitigated via explicit data augmentation or masked fine-tuning (Pan et al., 10 Oct 2025).
• Bottlenecks in sequential generation: Vanilla AR decoding is bounded by strictly sequential token emission and attention computation, but innovative techniques such as APAR, block diffusion, and linear attention substantially reduce these limitations (Liu et al., 2024, Wu et al., 30 Sep 2025, You et al., 2024).
• Hybridization requirements: Fusion approaches (e.g., MARIA) require two pretrained models with shared tokenizers, increasing resource consumption (Israel et al., 9 Feb 2025).
• Parameter scaling and alignment transfer: Optimal transfer in AR-MAP is sensitive to weight scaling factors; over-amplification or misalignment can sharply degrade performance (Lin et al., 2 Feb 2026).
• Architecture invariance for cross-modal and block-parallel models: Maintaining compatibility across AR, diffusion, and hybrid variants requires careful attention to attention-masking, cache management, and prompt structuring.
• Universality vs. practical usability: While AR-LLMs are theoretically universal, the programmatic specification, efficiency, and reliability of practical computation remain active areas of research (Schuurmans et al., 2024).

Broader research is focused on dynamic and richer hierarchies in APAR, adaptive inference block sizing, advanced cross-modal alignment, fine-grained task transfer, and hardware-aligned scaling of the AR paradigm. A plausible implication is that as architectural bottlenecks are relaxed via algorithmic innovations, AR-LLMs will remain central as universal sequence models for a diverse array of data modalities and reasoning tasks.