Structured State-Space Duality

• Structured State-Space Duality is a framework that establishes a precise equivalence between structured state-space models and masked attention mechanisms.
• It leverages semiseparable matrix structures to enable both linear-time recurrence and efficient batched masked attention for sequence transformations.
• The duality framework specifies necessary and sufficient conditions, guiding practical implementations in language modeling and sequential recommendation.

Structured State-Space Duality

Structured State-Space Duality (SSD) refers to a precise equivalence between a class of structured linear state-space models (SSMs) and masked self-attention mechanisms with specific matrix structure. In the context of sequence modeling and machine learning, particularly in architectures such as Mamba-2 and its derivatives, this duality provides two algorithmic realizations of the same sequence transformation: either as a linear-time recurrence (SSM) or as a masked attention (quadratic time in the naïve form, but admitting fast matrix decompositions). The SSD framework supports expressive yet efficient sequence models, deepens the algebraic link between recurrent and attention-based approaches, and underpins several recent state-of-the-art architectures in language modeling and sequential recommendation (Dao et al., 2024, Hu et al., 6 Oct 2025, Qu et al., 2024).

1. State-Space Models and Semiseparable Matrix Structure

The canonical discrete-time linear SSM is given by

$$x_{t+1} = A x_t + B u_t,\quad y_t = C x_t + D u_t,\qquad t=1,2,\ldots,T$$

where $A\in\mathbb{R}^{N\times N}$, $B\in\mathbb{R}^{N\times d_u}$, $C\in\mathbb{R}^{d_y\times N}$, $D\in\mathbb{R}^{d_y\times d_u}$. Unrolling yields a convolutional input-output map with kernel

$M_{j,i} = C A^{j-i-1} B,\qquad j>i$

The kernel matrix $M$ for an SSM is strictly lower-triangular and exhibits a special structure: for fixed $N$, any strictly lower-triangular submatrix of $M$ has rank at most $N$. Such matrices are called $N$-semiseparable or Sequentially Semiseparable (SSS), enabling algorithms for efficient matrix-vector multiplication in $O(TN)$ time (Dao et al., 2024, Hu et al., 6 Oct 2025).

2. Algebraic Duality: From SSM to Masked Attention

SSD arises by observing that, under restriction to scalar or diagonal state matrices $A$, the same kernel $M$ admits a masked attention realization. Specifically, in the scalar case $A = a I$, $M_{t,s}=a^{t-s}(C B)$, and the output can be computed as

$Y = (L \circ Q K^\top) U$

where $L_{t,s} = a^{t-s}$ for $t\geq s$ (and 0 otherwise) is a causal 1-semiseparable mask, $\circ$ denotes elementwise product, and both $Q$ and $K$ are vectors determined by $C$ and $B$. This equivalence generalizes to diagonal state matrices, where $M$ can be written as a sum of 1-semiseparable masks times rank-1 kernel matrices (Hu et al., 6 Oct 2025).

For general time-varying SSMs with token-wise parameter adaptation (as in Mamba and SSD4Rec), the SSM still produces an effective semi-separable structure, and duality holds at the level of masked linear attention (without softmax) (Dao et al., 2024, Qu et al., 2024).

3. Necessary and Sufficient Conditions, and Extensions

SSD provides both the algebraic sufficient and necessary conditions for which an SSM and a masked attention kernel coincide. The key criterion is the "new-column property": a kernel matrix $M$ can be represented as a 1-semiseparable mask times a low-rank kernel if and only if the number of linearly independent (new) columns does not exceed the SSM state dimension $N$ (Hu et al., 6 Oct 2025). This bridges expressive power and computational efficiency and guides architectural design:

• If $M$ admits at most $N$ new columns, SSM and masked attention are algorithmically interchangeable.
• If not, the SSM cannot be expressed as such attention, and the attention kernel may be of higher complexity or require more memory.

This duality fails for softmax attention: applying exponential normalization generally destroys low-rank or semiseparable structure, leading to rank explosion and precluding SSM equivalence (Hu et al., 6 Oct 2025).

4. Algorithmic Implications and Hardware-Efficient Realizations

SSD enables two complementary realization strategies:

• Recurrent (SSM) mode: Linear-time $O(TN)$, but sequential in time. Used in classical SSMs, Mamba-1, and in low-latency scenarios.
• Masked-Attention (matmul) mode: Quadratic-time $O(T^2 N)$ naïvely, but, using semiseparable/block-structured decompositions, can be batched and fused efficiently on GPU hardware. The SSD block decomposition in, e.g., Mamba-2 and SSD4Rec, partitions sequence and fuses block-matrix chains, substantially accelerating training and inference (Dao et al., 2024, Qu et al., 2024).

Hardware-aware SSD algorithms avoid sequential scans by expressing transformations as batched matrix products, which modern GPU libraries can execute with high throughput.

5. Applications: Language Modeling, Recommendation, and Beyond

SSD has catalyzed advances in language modeling and sequential recommendation:

• Mamba-2 and SSD4Rec: Leverage SSD to deliver state-of-the-art performance with favorable scalability. SSD4Rec, for example, addresses variable-length and long-range user history modeling by using "registers" to index user sequences and fuses block-matrix multiplications for batched processing (Qu et al., 2024).
• Time-Aware Modeling: TiM4Rec extends SSD with a time-aware structured mask, directly incorporating timestamp and time-interval information at the scalar mask level, restoring per-coordinate selectivity and further improving performance in sequential recommendation tasks (Fan et al., 2024).

Empirical benchmarks demonstrate that SSD-based architectures outperform both vanilla SSMs and Transformer baselines in wall-time vs. accuracy tradeoffs, memory, and scaling with sequence length (Dao et al., 2024, Qu et al., 2024, Fan et al., 2024).

6. Generalizations: Hamiltonian and Physical Inductive Biases

Recent work extends SSD to Hamiltonian dynamics (H-SSD), viewing the SSM hidden state as a phase-space point and evolving with energy-conserving symplectic integration. In Akasha 2, this paradigm incorporates Sparse Mixture of Hamiltonian Experts, Hamiltonian Flow Matching, and achieves spatiotemporal coherence and physical invariance in world models, leading to dramatic performance gains in video prediction and real-time visual synthesis (Meziani, 8 Jan 2026).

This suggests that SSD is a core algebraic mechanism, which, when combined with additional structure (e.g., symplecticity, mixture-of-expert routing), can encode rich inductive biases for domains with conservation laws or geometric symmetries.

7. Theoretical Significance and Limitations

SSD formalizes the algebraic continuum uniting state-space and attention mechanisms, characterizing precisely which sequence transformations can be realized efficiently by both approaches. The duality is exact only for linear SSMs and linear masked attention (without softmax). Nonlinearities (e.g., softmax, general outer-product kernels) usually break the equivalence. Nevertheless, the SSD framework underlies a class of models that simultaneously achieve linear-time inference and high parallelizability, with direct practical consequences for scalable modeling in modern deep learning (Hu et al., 6 Oct 2025, Dao et al., 2024).

In summary, Structured State-Space Duality underpins a foundational set of algorithms and architectures in efficient sequence modeling, providing a principled framework for bridging recurrent and attention-based computation, enabling hardware-optimized realization, and supporting ongoing innovations in both empirical performance and theoretical understanding.