Error-Free Linear Attention (EFLA)

• The paper introduces a novel analytic solution to the ODE governing linear attention, achieving error-free integration and improved noise robustness compared to prior methods.
• The methodology leverages a rank-1 matrix exponential identity to derive an infinite-order Runge–Kutta update, ensuring linear time and space complexity.
• Empirical evaluations demonstrate that EFLA outperforms frameworks like DeltaNet in language modeling accuracy and stability under noisy conditions without extra parameters.

Error-Free Linear Attention (EFLA) is a formulation of linear-time attention grounded in the exact solution of a continuous-time dynamical system. It provides a numerically stable, fully parallelizable generalization of the delta rule, enabling error-free integration while preserving linear time and space complexity. EFLA directly leverages the rank-1 structure of the attention dynamics, deriving an analytic solution via the matrix exponential and eliminating all discretization artifacts common in Euler- or Runge–Kutta-based updates. Empirical benchmarks demonstrate improved robustness under noise and enhanced language modeling accuracy over prior linear attention frameworks such as DeltaNet, without increasing parameter count (Lei et al., 14 Dec 2025).

1. Continuous-Time Dynamical System Foundation

The core of EFLA is the reinterpretation of the “delta-rule” linear attention update as an Euler discretization of a continuous-time ordinary differential equation (ODE). At time $t$, the system state is a matrix $\mathbf{S}(t)\in\mathbb{R}^{d\times d}$. The update is governed by a rank-1 dynamics matrix,

$\mathbf{A}(t) = \mathbf{k}(t)\,\mathbf{k}(t)^\top,$

and a forcing term,

$\mathbf{b}(t) = \mathbf{k}(t)\,\mathbf{v}(t)^\top.$

The ODE describes the evolution: $$\frac{d\mathbf{S}(t)}{dt} = -\mathbf{A}(t)\mathbf{S}(t) + \mathbf{b}(t),\quad \mathbf{S}(0) = \mathbf{0}.$$ In the discrete-time setting, the update corresponds to Euler integration: $$\mathbf{S}_{t} = (\mathbf{I} - \beta_t \mathbf{k}_{t}\mathbf{k}_{t}^\top)\mathbf{S}_{t-1} + \beta_t\mathbf{k}_{t}\mathbf{v}_{t}^\top,$$ where $\beta_t$ is a (possibly learnable) step-size. EFLA generalizes this by utilizing the exact analytic solution to the ODE, effectively subsuming and extending DeltaNet and related delta-rule approaches.

2. Analytic Solution via Matrix Exponential

For intervals $[t, t+\beta_t)$ where the dynamics are constant, the solution to

$$\dot{\mathbf{S}}(t) = -\mathbf{A}\mathbf{S}(t) + \mathbf{b}$$

with $\mathbf{A}$ rank-1 is

$$\mathbf{S}(t+\beta_t) = e^{-\mathbf{A}\beta_t}\mathbf{S}(t) + \int_0^{\beta_t} e^{-\mathbf{A}(\beta_t-s)}\,\mathbf{b}\ ds.$$

Exploiting the identity for a rank-1 matrix $\mathbf{A} = \mathbf{k}\mathbf{k}^\top$, with $\lambda = \mathbf{k}^\top\mathbf{k}$,

$$e^{-\beta_t \mathbf{k}\mathbf{k}^\top} = \mathbf{I} - \frac{1 - e^{-\beta_t \lambda}}{\lambda}\,\mathbf{k}\mathbf{k}^\top,$$

and observing that the integral reduces likewise, yields the closed-form update: $$\mathbf{S}_{t} = \left(\mathbf{I} - \frac{1-e^{-\beta_t\lambda_t}}{\lambda_t}\,\mathbf{k}_{t}\mathbf{k}_{t}^\top\right)\mathbf{S}_{t-1} + \frac{1-e^{-\beta_t\lambda_t}}{\lambda_t}\,\mathbf{k}_{t}\mathbf{v}_{t}^\top.$$ This update is the infinite-order Runge–Kutta (RK$_\infty$) limit of Taylor-series discretizations, exactly integrating the continuous dynamics at each step (Lei et al., 14 Dec 2025).

3. Computational Properties and Parallelization

EFLA inherits linear-time complexity per step due to the rank-1 outer products and matrix–vector computations. Each token step requires $O(d^2)$ operations, and for sequence length $L$, total complexity is $O(Ld^2)$. The memory footprint consists solely of the current $\mathbf{S}_{t-1}$ and current $(\mathbf{k}_t, \mathbf{v}_t)$, independent of $L$.

Full sequence parallelism is attainable via WY-representation and parallel prefix (scan) methods over the sequence of update matrices $$(\mathbf{I} - \alpha_t \mathbf{k}_t\mathbf{k}_t^\top)$$, enabling chunked computation and reducing serial depth to $O(L/C + \log C)$ for chunk size $C$. The analytic update ensures no numerical error is introduced per step, entirely eliminating discretization error from the time integration (Lei et al., 14 Dec 2025).

4. Algorithmic Workflow

The high-level algorithm for a single-head EFLA is as follows (notations as above):

Input: sequence {(q_t, k_t, v_t)}_{t=1}^L, step sizes {β_t} Initialize S ← zero matrix in R^{d×d} for t in 1…L: λ ← k_t^T k_t # scalar if λ > ε: α ← (1 - exp(-β_t * λ)) / λ else: α ← β_t # fallback for small λ # decay old state along k_t δS ← k_t * ( (k_t^T S) ) S ← S - α * δS # inject new key-value S ← S + α * (k_t v_t^T) o_t ← S^T q_t # output at time t end for Output: sequence {o_t}

In practice, implementations utilize multi-head variants and apply parallel prefix computation for efficiency at scale. Batched execution is facilitated through chunking without additional serial bottlenecks (Lei et al., 14 Dec 2025).

5. Empirical Evaluation

Numerical Stability: In systematic experiments on sMNIST with extreme input scaling and noise, EFLA demonstrates robustness where DeltaNet fails—maintaining >90% accuracy under scale=10 and severe noise/distortions, while DeltaNet degrades to random-guessing levels (<20% at similar conditions).

Language Modeling Benchmarks: On standard 340M-parameter models trained on WikiText and LAMBADA:

• WikiText perplexity: DeltaNet 38.09 → EFLA 37.01 (↓~3%)
• LAMBADA perplexity: 96.26 → 81.28; accuracy: 22.5% → 23.9%
• On zero-shot tasks (PIQA, HellaSwag, BoolQ, etc.): accuracy improves from 40.9% to 42.6%; +7.4 points on BoolQ.
• For 1.3B-parameter models (16B tokens): LAMBADA perplexity improves from 69.27 → 61.83; WikiText from 31.82 → 31.48; average task accuracy 43.0% → 43.9%.

Noise Performance: Across sMNIST variants (dropout, additive Gaussian, OOD scaling), EFLA consistently outperforms across all noise regimes, confirming that error-free ODE integration achieves higher-fidelity memory in practice (Lei et al., 14 Dec 2025).

6. Theoretical Considerations, Extensions, and Limitations

Error-Free Guarantee: The analytic update’s exactness depends on two properties:

1. The closed-form for linear ODEs via matrix exponentials.
2. The reduction of matrix exponential computation from $O(d^3)$ to $O(d^2)$ via the rank-1 identity.

Relation to Runge–Kutta: The EFLA update is the infinite-order (RK$_\infty$) Taylor expansion; truncation recovers conventional finite-order Runge–Kutta discretizations, including the first-order delta rule.

Extensions: Generalization to multi-rank dynamics ($A = \sum_{i=1}^r u_i v_i^\top$) is contingent on efficient low-rank matrix exponential computation. For general (full-rank) $A$, efficient approximations are required (e.g., bilinear transforms as in SSMs).

Limitations:

• Applicability is currently restricted to settings where the dynamics matrix $A$ is exactly rank-1 at every token.
• In high-dimensional settings absent normalization, $\alpha_t<1$ may reduce convergence rates, necessitating larger learning rates.
• Extension to error-free integration across multiple heads/layers and in architectures featuring cross-attention remains open.

In summary, EFLA provides an exact, parameter-free, error-free update mechanism for linear attention, enabling robust and scalable long-context modeling by eliminating discretization artifacts and preserving full parallelism and linear complexity (Lei et al., 14 Dec 2025).