Matrix Memory LSTMs (xLSTM)

Updated 17 January 2026

Matrix Memory LSTMs (xLSTM) are recurrent architectures that generalize standard LSTMs by replacing scalar cell states with matrix-valued memory and employing exponential gating for stable, non-saturating updates.
They utilize covariance-style memory updates and efficient parallel scan algorithms to achieve linear scaling with sequence length, enhancing training speed and long-range dependency capture.
xLSTM models demonstrate competitive performance in language modeling, time series forecasting, vision, and trajectory prediction, offering practical advantages over traditional LSTMs and Transformers.

Matrix Memory LSTMs (xLSTM) are a family of recurrent architectures that generalize Long Short-Term Memory networks by introducing matrix-valued memory and often exponential gating mechanisms. This design dramatically increases the representational capacity and computational efficiency for sequential, spatial, and spatiotemporal modeling. xLSTM variants have demonstrated competitive results across domains including language modeling, long-term time-series forecasting, trajectory prediction, and vision tasks, particularly as efficient alternatives to Transformers for modeling long-range dependencies with linear computational scaling (Beck et al., 2024, Beck et al., 2 Oct 2025, Dutta et al., 2024, Alkin et al., 2024, Huang et al., 2024).

1. Mathematical Formulation and Core Architectural Innovations

Matrix-memory LSTM (“mLSTM”) cells depart from standard LSTM design by replacing the scalar cell state $c_t$ with a matrix $M_t\in\mathbb{R}^{d\times d}$ , updated by a covariance-style formula. Gate units may utilize strictly positive exponential activations for the input and forget gates, resulting in non-saturating, multiplicative gating flows.

A canonical mLSTM cell computes, at time step $t$ :

Input projections:

$k_t = W_k x_t + b_k \in \mathbb{R}^d$

$v_t = W_v x_t + b_v \in \mathbb{R}^d$

$q_t = W_q x_t + b_q \in \mathbb{R}^d$

Gates:

$i_t = \exp(\tilde{i}_t),\quad f_t = \exp(\tilde{f}_t),\quad o_t = \sigma(\tilde{o}_t)$

(in some variants, $f_t$ may use sigmoid).

Matrix-memory update:

$M_t = f_t \odot M_{t-1} + i_t \odot (v_t k_t^{\top})$

(broadcast elementwise).

Output read:

$h_t = o_t \odot \mathrm{LN}(M_t q_t)$

where layer or RMS normalization may be used.

A normalization accumulator $n_t$ is often maintained to stabilize long-sequence scale, and a max-state $m_t$ is used to renormalize exponentials (Beck et al., 2024, Alharthi et al., 2024).

Key architectural advances include:

Matrix-memory structure: $M_t$ can store second-order statistics of key/value pairs, enabling associative recall and non-Markovian information storage (Beck et al., 2024).
Exponential (or log-space stabilized) gating: Exp-gated input and forget units offer non-saturating, revisable gating and sustained gradient flow across long contexts (Beck et al., 2024, Alharthi et al., 2024).
Parallel/chunked scan computation: Covariance-style recurrences allow for $O(\log T)$ parallel scan implementations, enabling practical scaling to very long sequence lengths (Alkin et al., 2024, Beck et al., 18 Mar 2025).
Block-multihead recurrence: For efficiency, mixing matrices (e.g., $R$ in sLSTM) may be block-diagonal across multiple heads (Kraus et al., 2024).

2. Algorithmic and Implementation Details

The memory update in mLSTM can be summarized by the following procedural steps (Beck et al., 2024, Beck et al., 18 Mar 2025):

def mLSTM_step(x_t, h_prev, M_prev, n_prev, m_prev):
    # Project input
    k_t = W_k @ x_t + b_k
    v_t = W_v @ x_t + b_v
    q_t = W_q @ x_t + b_q

    # Compute gates
    i_t = exp(W_i @ x_t + r_i @ h_prev + b_i)
    f_t = exp(W_f @ x_t + r_f @ h_prev + b_f)
    o_t = sigmoid(W_o @ x_t + r_o @ h_prev + b_o)

    # Log-space stabilization for long sequences
    m_t = max(log(f_t) + m_prev, log(i_t))
    i_dash = exp(log(i_t) - m_t)
    f_dash = exp(log(f_t) + m_prev - m_t)

    # Memory update
    M_t = f_dash * M_prev + i_dash * np.outer(v_t, k_t)
    n_t = f_t * n_prev + i_t * k_t

    # Output computation
    h_raw = M_t @ q_t / max(abs(n_t @ q_t), 1)
    h_t = o_t * h_raw
    return h_t, M_t, n_t, m_t

Efficient GPU kernels exploit chunkwise parallel computation and intra-chunk tiling, as seen in Tiled Flash Linear Attention (TFLA), which can outperform FlashAttention and Mamba for long-context model training and inference (Beck et al., 18 Mar 2025). Two-level tiling allows arbitrarily large chunk sizes and minimal memory footprint per chunk.

3. Integration into Deep Backbone Architectures

Matrix-memory xLSTM blocks can be residually stacked, analogous to Transformer architectures. In vision, xLSTM serves as a generic backbone for patch-token processing, alternating scan directions to capture both forward and backward context, and allowing layer-wise parallelization (Alkin et al., 2024, Huang et al., 2024). For 3D medical image segmentation, Vision-xLSTM cells are incorporated into U-Net encoders and decoders, connecting convolutional feature maps to global context memory at each depth (Dutta et al., 2024).

Typical backbone integration involves:

Residual block construction: Each xLSTM cell is wrapped with normalization and MLP layers, forming repeatable blocks.
Scan direction alternation: Odd layers process tokens in raster order, even layers in reverse, yielding bi-directional context (Alkin et al., 2024).
Cluster-masked prediction: In large vision models (MAL), clusters of spatially adjacent patches are grouped and masked for efficient local context learning (Huang et al., 2024).
Domain-specific embedding and projection: For timeseries and trajectory tasks, domain-specific preprocessing (trend/seasonal splits, kinematic projection) is applied prior to xLSTM processing (Alharthi et al., 2024, Chugh et al., 31 Oct 2025).

4. Computational Complexity, Scaling Laws, and Efficiency

The hallmark of matrix-memory xLSTM is linear context scaling. Each memory update is $O(d^2)$ per step, and overall network cost scales as $O(L d^2)$ rather than the $O(L^2 d)$ in attention-based Transformers for sequence length $L$ and hidden dimension $d$ (Beck et al., 2 Oct 2025). Efficient parallel computation is achieved via scan algorithms and chunked kernels (TFLA), leading to practical throughput advantages at long contexts.

Comparative scaling law analyses reveal:

Model	Context Scaling	Training Loss Exponent ( $\alpha$ )	Inference Step Time
Transformer	$O(L^2)$	0.53	$\propto L$ or $O(L^2)$ (prefill)
xLSTM	$O(L)$	0.73	$O(1)$ (generation)

xLSTM exhibits steeper scaling exponents, implying superior returns to increasing parameters or data volume. The compute-optimal model size degrades slowly with context length, making xLSTM preferable for tasks involving large contexts ( $L > 8$ K) (Beck et al., 2 Oct 2025).

5. Empirical Performance Across Domains

xLSTM models have achieved competitive or state-of-the-art results in:

Language modeling: Lower perplexity than Transformers, Mamba, RWKV, and state-space models across model sizes (125M–2.7B), robust to extrapolation at 16K context (Beck et al., 2024, Beck et al., 2 Oct 2025).
Time series forecasting: In xLSTMTime and xLSTM-Mixer, consistent gains in MSE and MAE on electricity, weather, and traffic datasets vs. DLinear, PatchTST, and TimeMixer baselines (Alharthi et al., 2024, Kraus et al., 2024).
Vision: On ImageNet-1K, MAL xLSTM models outperform comparable transformer baselines, showing robust transfer and strong semantic segmentation performance, especially when paired with cluster-masked, multi-task pretraining (Huang et al., 2024). Vision-xLSTM U-Nets improve Dice and IoU by 1–2 points over plain U-Net and ConvLSTM baselines on Synapse, ISIC, and ACDC segmentation (Dutta et al., 2024).
Trajectory prediction: Physics-aware X-TRACK (xLSTM encoder with kinematic postprocessing) yields physically feasible vehicle trajectories and outperforms standard LSTM and attention-based baselines on highD and NGSIM (Chugh et al., 31 Oct 2025).

Matrix-memory xLSTM provides several advantages over conventional LSTM and Transformer layers:

Memory capacity: $O(d^2)$ via matrix-memory vs. $O(d)$ in vector LSTM; richer associative retrieval and temporal mixing (Beck et al., 2024).
Gating and gradient flow: Exponential gates allow non-saturating updates, revisability, and more stable training for long sequences (Beck et al., 2024).
Efficiency at large contexts: Linear scaling of compute and inference time yields practical speed, especially with optimized kernels (TFLA) (Beck et al., 18 Mar 2025).
No need for full past-key-value caches: Forward recurrence obviates attention cache growth (Beck et al., 2 Oct 2025).

Ablation studies confirm that removing matrix memory (reverting to ConvLSTM) or replacing with a vector memory degrades empirical performance (Dutta et al., 2024); cluster-based masking further boosts local feature extraction (Huang et al., 2024).

7. Open Issues, Extensions, and Future Outlook

Potential directions include:

Learnable cluster masking and content grouping: Replacing fixed-grid clustering with data-driven block segmentation (Huang et al., 2024).
Integration with spatial attention or windowed modules: Hybrid models to address very high-resolution or irregular spatial inputs.
Vision and multimodal pretraining: Robust transfer observed in MAL with multi-task pretraining; extensions to video via temporal cluster recurrence suggested (Huang et al., 2024).
Kernel optimization: Further refinement of TFLA and mLSTMsig variants for reduced memory and faster training on large GPU clusters (Beck et al., 18 Mar 2025).

Limitations include potential underemphasis of high-frequency details in MSE-based training, and suboptimality of fixed clustering for highly irregular data. Nevertheless, empirical evidence points to matrix-memory xLSTM as a scalable, high-capacity alternative for long-context modeling across modalities (Beck et al., 2024, Beck et al., 2 Oct 2025, Huang et al., 2024).