Linear Recurrent Units for Efficient Sequence Modeling

• Linear Recurrent Units (LRU) are neural network layers defined by diagonal complex recurrences that model long-range dependencies efficiently.
• Their design enables full-sequence parallel computation using scan algorithms, drastically reducing training time compared to standard RNNs.
• Hybrid extensions and gating mechanisms enhance LRU expressivity, optimizing performance in applications like sequential recommendation and time series forecasting.

Linear Recurrent Units (LRU) refer to a class of neural network layers characterized by a purely linear and (typically complex-diagonal) hidden-to-hidden recurrence, enabling efficient modeling of long-range dependencies in sequential data. LRUs are designed to retain the forward-incremental processing property of classic RNNs while admitting highly parallel “prefix-scan” or convolutional formulations for efficient training. They have become foundational for recent advances in large-scale sequence modeling, especially in domains where traditional RNNs suffer from optimization or scalability limitations, and attention mechanisms (as in Transformers) are prohibitively expensive for long sequences.

1. Mathematical Formulation and Parameterization

The fundamental LRU recurrence operates on a hidden state $h_k \in \mathbb{C}^N$ and input $x_k \in \mathbb{R}^{H}$:

$h_k = \Lambda h_{k-1} + \Gamma B x_k$

$y_k = \Re(C h_k) + D x_k$

Here:

• $\Lambda = \mathrm{diag}(\lambda_1, ..., \lambda_N)$ is a diagonal matrix of learned complex eigenvalues. Each eigenvalue is parameterized as $\lambda_i = e^{-\nu_i + j\theta_i}$, with $\nu_i > 0$ enforcing $|\lambda_i| < 1$ for stability (Orvieto et al., 2023, Yue et al., 2023, Liu et al., 11 Apr 2025).
• $B \in \mathbb{C}^{N \times H}$ and $C \in \mathbb{C}^{H \times N}$ are input and output projection matrices, respectively.
• $\Gamma = \mathrm{diag}(e^{\gamma})$ is an optional per-component input normalization vector.
• $D \in \mathbb{R}^{H \times H}$ defines a possible residual skip from input to output.

Some implementations employ additional gating (Liu et al., 2024) or input-dependent parameters, but the above fixed-coefficient linear structure is central to the canonical LRU.

By diagonalizing the transition operator $A$ in the complex domain ($A = P \Lambda P^{-1}$), the recurrence over length-$L$ sequences is analytically solvable via:

$h_k = \sum_{j=0}^{k-1} \Lambda^j \Gamma B x_{k-j}$

This structure enables efficient associative scan algorithms and supports O($\log L$) training parallelism (Yue et al., 2023, Orvieto et al., 2023).

2. Efficient Parallelism and Scalability

Unlike standard RNNs, whose sequential hidden-state updates are an impediment to high-throughput parallel training, the purely linear and diagonalized form of LRUs admits full-sequence parallel computation. Specifically, one can:

• Employ divide-and-conquer scan algorithms (Blelloch scan/up-sweep/down-sweep) to compute $h_{1..L}$ in O($\log L$) steps with O($L$) work (Yue et al., 2023, Liu et al., 11 Apr 2025, Liu et al., 2024).
• Implement FFT-based convolution approaches for certain cases, exploiting the analyticity of the recurrence for further speedup (Katsch, 2023).

The computational benefits can be summarized as follows:

Model	Training Time per Epoch	Inference per Step	Space Complexity
Standard RNN	$O(L H^2)$	$O(H^2)$	$O(H^2)$
Transformer	$O(L^2 H + L H^2)$	$O(L H^2)$	$O(L H + H^2)$
LRU (parallel)	$O(\log L H^2)$	$O(H^2)$	$O(|I| H + H^2)$

Where $L$ is sequence length, $H$ hidden size, $|I|$ vocabulary cardinality (Yue et al., 2023).

This architecture allows LRUs to achieve “RNN-like” constant-time online inference and “Transformer-style” highly parallel training in the same model (Yue et al., 2023, Liu et al., 11 Apr 2025, Orvieto et al., 2023).

3. Nonlinear Extensions and Hybrid Architectures

A strictly linear recurrence may underfit in practice. To address this, LRUs are typically embedded within deep module stacks with nonlinearity, normalization, and residual connectivity:

• LayerNorm or BatchNorm is applied post-recurrence for training stability (Yue et al., 2023, Orvieto et al., 2023, Ling et al., 2 Feb 2026).
• Pointwise position-wise feed-forward networks (PFFN) using GELU or GLU offer expressivity.
• Residual skip connections facilitate gradient propagation in deep architectures (Orvieto et al., 2023, Ling et al., 2 Feb 2026).

Variants such as Behavior-Dependent LRUs (BD-LRU) (Liu et al., 2024) or Recurrent Trace Units (RTUs) (Elelimy et al., 2024) incorporate input-dependent gates or simple nonlinearities inside the recurrence, improving selective memory and sample efficiency.

Bidirectional extensions, such as BLUR, execute forward and backward LRUs in parallel, merging their outputs for bidirectional context modeling (Liu et al., 11 Apr 2025).

4. Theoretical Properties and Model Capacity

Despite the absence of nonlinear hidden-to-hidden transitions, LRUs are universal approximators of finite sequential functions when followed by sufficient output capacity. For example, linear recurrent networks can interpolate any target sequence $f(0), ..., f(n)$ with $N_h \geq n-d$ hidden units ($d$ is output dimension), and the optimal output weights are analytically computed as a single pseudo-inverse or least-squares solve (Stolzenburg et al., 2018).

Spectral structure is central: the eigenvalues of the diagonal transition matrix $\Lambda$ determine the unit’s memory horizon and oscillatory/decay behavior. Spectral radius $|\lambda| < 1$ ensures stability and precludes gradient explosion or vanishing (Liu et al., 11 Apr 2025, Orvieto et al., 2023, Stolzenburg et al., 2018). Pruning of spectral components yields model compression with minimal loss, with long-run hidden trajectories converging to ellipses or fixed points dictated by dominant eigenvalues (Stolzenburg et al., 2018).

5. Comparison to Other Sequence Models

LRUs are closely related to structured state-space models (S4, S5, DSS), which discretize continuous-time diagonal dynamics for similar benefits. LRU is fully discrete, does not require ODE solvers or HiPPO initialization, and offers direct parameterization and initialization control (Orvieto et al., 2023, Liu et al., 11 Apr 2025).

Relative to attention-based models, LRUs:

• Avoid the quadratic cost in sequence length of self-attention,
• Eliminate the need for caching all past hidden states (no key-value bottleneck),
• Achieve order-of-magnitude faster inference and training throughput on long sequences (Yue et al., 2023, Liu et al., 2024, Liu et al., 11 Apr 2025).

However, vanilla LRU may lack the flexible, data-dependent context integration afforded by dynamic attention or gate-controlled recurrences (as in GateLoop (Katsch, 2023)). Extensions that add data-controlled gates (e.g., BD-LRU, GateLoop) produce substantial performance gains, especially on tasks with variable dependency patterns (Liu et al., 2024, Katsch, 2023).

6. Empirical Results and Applications

LRU-based architectures have achieved state-of-the-art or highly competitive results on diverse benchmarks:

• Sequential Recommendation: LRURec and BD-LRU outperform self-attention models (SASRec, BERT4Rec) and recurrent baselines (GRU4Rec) by 4–17% relative Recall@10/20 on MovieLens-1M, Amazon Beauty, Steam, and XLong datasets, with 5–10× higher per-request inference throughput for long user histories (Yue et al., 2023, Liu et al., 2024).
• Long-Range Sequence Modeling: On Long Range Arena benchmarks (sCIFAR, ListOps, IMDB, sMNIST), LRU matches or exceeds SSMs and trains much faster than nonlinear RNNs (Orvieto et al., 2023, Liu et al., 11 Apr 2025).
• Time Series Forecasting: BLUR (bidirectional LRU) achieves the lowest MAEs in 40/50 tasks against LRU, S4, Informer, with $\sim$3× lower training and inference time than S4/S5 and far below Transformer costs (Liu et al., 11 Apr 2025).
• Reinforcement Learning: LRUs allow exact RTRL updates in O($n$) (vs O($n^3$) for generic RNNs), and RTUs further improve learning efficiency, stability, and return in partially observable settings (e.g., Mujoco P/V, POPGym) (Elelimy et al., 2024).
• Handwriting Recognition: The SW-PS+LRU framework achieves state-of-the-art accuracy and rapid convergence on rotation-augmented handwritten character data, substantially exceeding convolutional and Transformer baselines (Ling et al., 2 Feb 2026).
• Function Approximation: LRUs interpolate arbitrary sample sequences, outperform LSTM and echo state networks on tasks such as multi-frequency signal prediction, and enable architecture compression via spectral analysis (Stolzenburg et al., 2018).

7. Limitations, Generalization, and Future Extensions

The canonical LRU, by itself, cannot adapt its memory scale or input selection based on data context. Data-controlled and input-gated variants (GateLoop, BD-LRU) are required to unlock full sequence modeling power, as the ability to modulate forgetting and retention per step is crucial for high-performing models on real-world tasks (Katsch, 2023, Liu et al., 2024). Further limitations include:

• Requirement for diagonalizability of the recurrence (and restriction to the complex field),
• Potential underfitting if nonlinearity or gating is omitted,
• Custom kernel implementations for efficient parallel scan with input-dependent gates (Liu et al., 2024).

Potential research directions include principled multi-layer RTRL for non-linear extensions, development of more expressive merging or output heads, and hybridization of LRU layers with local/global attention modules (Liu et al., 11 Apr 2025, Elelimy et al., 2024).

---

In sum, Linear Recurrent Units define a tractable, highly efficient, and empirically competitive paradigm for sequential modeling, leveraging complex-diagonal recurrences, analytic solutions, and hardware-accelerated parallelization. Their flexibility as both a direct online RNN and a backend for large-scale batch training has established LRU-based models as a central tool in high-performance sequence learning (Orvieto et al., 2023, Yue et al., 2023, Liu et al., 11 Apr 2025, Liu et al., 2024, Katsch, 2023).