Deep State Space Models Overview

• Deep state space models are architectures that combine classical state-space principles with neural networks to capture nonlinear sequence dynamics.
• They integrate stochastic and deterministic modeling, enabling uncertainty quantification and expressiveness that rivals transformer-based approaches.
• Selective variants introduce content-dependent state updates to efficiently capture long-range dependencies and improve convergence in sequence tasks.

Deep state space models (DSSMs) constitute a broad and rapidly advancing class of architectures that generalize classical state-space frameworks using deep learning methods. They form a unifying technical foundation for sequence modeling across domains such as language, vision, time series forecasting, control, and system identification. By integrating both stochastic and deterministic mechanisms—often parameterized by neural networks—deep SSMs efficiently capture long-range dependencies, allow for uncertainty quantification, and, in their "selective" variants, attain expressiveness on par with or surpassing transformer-based models.

1. Definition and Fundamental Architecture

A deep state space model is characterized by a latent state process $\{z_t\}$, whose evolution and observation are both parameterized by highly flexible (often nonlinear) neural mappings. The canonical discrete-time setup takes the form: $$z_0 \sim p_\theta(z_0), \quad z_t \mid z_{t-1} \sim p_\theta(z_t \mid z_{t-1}), \quad x_t \mid z_t \sim p_\theta(x_t \mid z_t)$$ where $p_\theta$ denotes families of transition and emission densities, frequently Gaussian with means and covariances output by neural networks (Lin et al., 2024, Gedon et al., 2020).

Two core architectural threads underlie "deep" state-space modeling:

• Recursive nonlinearity: Transition $z_t = f_\theta(z_{t-1}, u_t)$ and emission $x_t = g_\theta(z_t)$ typically depend on neural networks, capturing complex, nonlinear dependencies not possible in linear-Gaussian SSMs (Gedon et al., 2020).
• Variational inference and amortized posteriors: Learning relies on variational autoencoder (VAE) methodologies, where amortized inference networks $q_\phi(z_{1:T}|x_{1:T})$ approximate intractable posteriors, enabling gradient-based optimization on a structured ELBO (Lin et al., 2024, Wu et al., 2022, Li et al., 2021).

Deep SSMs appear in many distinctive contexts: as generative time series models (Schmidt et al., 2018), stochastic dynamical system identifiers (Gedon et al., 2020), probabilistic sequence predictors (Li et al., 2021), latent process forecasters in knowledge tracing (Christie et al., 2024), and as building blocks for scalable sequence-processing layers in foundation models (Cirone et al., 2024, Schaller et al., 17 Nov 2025).

2. Selective State Space Models: Theoretical Foundations and Dynamics

Selective state space models (selective SSMs), including Mamba (S6), Gated Linear Attention (GLA), S4, and GateLoop, have established a new paradigm in sequence modeling by incorporating input-dependent multiplicative interactions ("selectivity") in their state-updates. The S6/Mamba recurrence, for instance, augments a standard linear recurrence with a data-controlled update: $$h_{t} = \exp(\Delta_t A)h_{t-1} + \Delta_t B x_t, \quad \Delta_t = \mathrm{softplus}(\alpha^\top x_t + \beta)$$ where $A$ is typically diagonal and $\Delta_t$ serves as a channel-wise content-dependent step size (Vo et al., 2024, Cirone et al., 2024).

Theoretical analysis shows that such selective architectures can be framed as random projections of path signatures in the sense of rough path theory. The hidden state $Z_t$ is, with high probability, a low-dimensional linear projection of the signature $\operatorname{Sig}(X)_{0,t}$—the universal basis for nonlinear, multi-timescale interactions between tokens (Cirone et al., 2024). This signature expansion includes all iterated integrals of the input path, giving selective SSMs the ability to represent high-order, non-local dependencies efficiently.

Key theorem: For selective SSMs with random (Gaussian) recurrences and content-gated transitions, any continuous functional of the path signature can be approximated arbitrarily well by a suitable linear read-out from the hidden state, provided the state dimension $N$ is large enough. This probabilistically guarantees universal approximation power analogous to transformers' tokenwise attention, but achievable at linear time and memory complexity (Cirone et al., 2024).

3. Token Dynamics, Stability, and Regularization Mechanisms

Recent work has characterized the dynamical system induced by selective SSMs in the continuous-time limit and delineated two fundamentally different asymptotic scenarios for token evolution (Vo et al., 2024):

• Convergence regime $(\mu < 0)$: Tokens collapse to zero, leading to degeneration of the model's expressivity; attention weights decay as $O(1/t)$. This regime is associated with negative eigenvalues of the input-output form $\mu = S_C^\top S_B$.
• Divergence regime $(\mu > 0)$: Tokens diverge, either slowly (logarithmic growth at $O((\ln t)^l)$ per token) or, if the step-size signs permit, blow up in finite time. Divergence maintains model activity, and different tokens will contribute unequally to the updates, fundamentally altering learning dynamics.

Empirical evidence shows that the convergent scenario increases perplexity and harms predictive performance (e.g., WikiText-103 test PPL $\sim$17.26 vs. 16.71 for positive-definite $\mu$). Enforcing positive-definiteness in $S_C^\top S_B$ (e.g., via $LDL^T$ reparametrization) prevents collapse and leads to faster and stronger convergence (Vo et al., 2024).

A further refinement uses token reordering based on a learnable importance score: by soft-permuting tokens with higher "importance" into positions where state growth is fastest, gradient flow is preserved and the model attains superior convergence and accuracy (e.g., ImageNet-1K MambaVision-T top-1 rises from 81.90% to 82.02%) (Vo et al., 2024).

4. Broader Modeling Landscape and Methodological Variants

The landscape of deep SSMs encompasses a wide range of modeling techniques, from deterministic and stochastic nonlinear transitions to graph-based latent state evolution:

• Stochastic DSSMs: Models such as those in (Schmidt et al., 2018, Li et al., 2021, Liu et al., 2023, Wu et al., 2022) introduce latent stochastic states, probabilistic emissions, and flow-based or GP-based transitions. Inference is typically handled by amortized variational filtering or particle-based methods.
• Gated/Graph-SSMs: GDSSM (Look et al., 2023) extends deep SSMs to multi-agent, interacting dynamical systems via graph neural networks, employing sample-free, deterministic moment-matching to propagate multimodal uncertainty in a scalable manner.
• Pruning and Efficiency: LAST (Gwak et al., 2024) implements structured, layer-adaptive pruning based on the $\mathcal{H}_\infty$ norm of SSM state subsystems, achieving up to one-third state reduction with negligible performance loss, by leveraging modal truncation theory for deep diagonal SSMs.
• Bidirectional and Cross-Time Interactions: Naga (Schaller et al., 17 Nov 2025) introduces a Vedic encoding with bidirectional (forward and reverse) Hadamard interaction, capturing long-range, second-order cross-time dependencies in a lightweight and interpretable way.

5. Interpretability, Uncertainty Quantification, and Applications

A distinctive feature of deep SSMs is their capacity for principled interpretability and uncertainty assessment. Approaches such as Dynamic LENS (Christie et al., 2024) preserve full epistemic uncertainty via high-dimensional Gaussian posteriors propagated through analytic Kalman-style updates, in contrast to standard deep knowledge tracing methods that only track point estimates.

Interpretability can be further enhanced by architectural choices such as linear output decoders (mapping latent variables directly to observations), shrinkage priors for sparsity and robustness (as in (Wu et al., 2022)), and ARD subnetworks for feature relevance determination (as in (Li et al., 2021); see learned $w$ weights ranking exogenous variables by input importance).

Applications of deep SSMs span:

• Language modeling and text generation (Schmidt et al., 2018, Vo et al., 2024)
• Long-range time-series forecasting and system identification (Gedon et al., 2020, Li et al., 2021, Gwak et al., 2024)
• Multi-agent trajectory prediction (Look et al., 2023)
• Vision (image classification/detection) (Liu et al., 12 Feb 2025, Vo et al., 2024)
• Reinforcement learning and model-based control, including epistemic and aleatoric uncertainty decomposition (Becker et al., 2022)

6. Theoretical Analysis of Learning Dynamics

The learning principles governing deep SSMs—especially their convergence and parameter evolution during training—are becoming clearer due to analytical developments using frequency-domain analysis and connections to deep linear networks (Smékal et al., 2024):

• For linear SSMs, convergence rates improve with latent state dimension and favorable data covariance spectra; the learning dynamics diagonalize in frequency, and over-parameterization guarantees faster convergence.
• The equivalence between 1D SSMs and two-layer deep linear networks allows transfer of insights from classic deep learning theory (e.g., Saxe's results).
• In selective SSMs, multiplicative gating and self-attention mechanisms provide expressive, inductive bias enabling gradient-descent-like in-context learning, directly linking SSMs to transformer-style sequence learners (Sushma et al., 2024, Cirone et al., 2024).
• Major analytical challenges remain in the nonlinear, multi-layer regime due to non-commutativity and frequency coupling, but perturbative or random-feature-based extensions are outlined as promising future directions (Smékal et al., 2024).

7. Trends, Limitations, and Future Directions

While deep SSMs are now state-of-the-art for many sequence modeling benchmarks, several challenges and prospects merit attention:

• Inference and expressivity: Posterior collapse and underestimation of temporal uncertainty remain risks; richer inference architectures and β-VAEs are under study (Lin et al., 2024).
• Identifiability and generalization: Nonlinear SSMs are non-identifiable up to diffeomorphism; advances in regularization and structure learning are needed.
• Scaling and efficiency: SSM layers (S4, S5, Mamba, etc.) offer linear-time solutions for very long sequences and efficient state propagation, but efficient non-diagonal dynamics, pruning, and cross-modal fusion require further work (Gwak et al., 2024, Schaller et al., 17 Nov 2025, Liu et al., 12 Feb 2025).
• Continuous-time and irregular sampling: Latent neural ODE/SDE approaches show promise for mixed-frequency and irregularly spaced data, and are naturally suited to bridging discrete and continuous sequence modeling (Lin et al., 2024).
• Broader applicability: Hybrid schemes combining classical filtering with deep, learned dynamics, integration into LLMs, graph neural SSMs, and advanced system-theoretic regularization for robust control are active research frontiers.

In summary, deep state space models provide a theoretically sound, computationally scalable, and highly expressive toolkit for modern sequence modeling, both as probabilistic generative models and as efficient, replacement layers for transformer-like architectures. Developments in token dynamics, expressive selectivity, theoretical guarantees, and practical innovations continue to drive the field, enabling broad and high-fidelity deployment across domains (Lin et al., 2024, Vo et al., 2024, Cirone et al., 2024, Schaller et al., 17 Nov 2025).