DA-SHRED: Shallow Recurrent Decoder Assimilation

• The paper presents a latent assimilation framework that compresses high-dimensional states into a low-dimensional space for real-time reconstruction.
• It combines a shallow encoder-decoder with a recurrent model and Kalman-style updates to integrate sparse sensor data and simulation proxies.
• Sparse regression (SINDy) identifies missing dynamical terms, achieving a significant reduction in RMSE and bridging the SIM2REAL gap.

Data Assimilation with a SHallow REcurrent Decoder (DA-SHRED) is a machine learning framework designed to integrate sparse sensor data with computational simulation models for high-dimensional, spatiotemporal physical systems. It operates by embedding the full system state into a low-dimensional latent space, enabling real-time reconstruction and discrepancy modeling between model predictions and experimental measurements. The methodology addresses the simulation-to-real (SIM2REAL) gap introduced by unmodeled physics and parameter misspecification, providing both assimilation and identification of missing dynamics through sparse-regression in the latent space (Bao et al., 1 Dec 2025).

1. Problem Formulation and Mathematical Framework

DA-SHRED considers a high-dimensional system state $x_t \in \mathbb{R}^n$ evolving under unknown real physics. Available resources are sparse point-sensor measurements $y_t \in \mathbb{R}^p$ and a reduced simulation proxy $N$ that approximates the true system dynamics, $\,\dot{x} = N(x, t)\,$. Observations are modeled as $\,y_t = H x_t + \eta_t\,$, with $H \in \mathbb{R}^{p \times n}$ a known linear observation operator and $\eta_t$ measurement noise.

The dual objectives are:

• Assimilate incoming measurements $y_t$ into a reduced latent representation $z_t \in \mathbb{R}^r\,\,(r \ll n)$ to reconstruct the full state $\hat{x}_t \approx x_t$ in real time.
• Discover missing or unmodeled dynamics $y_t \in \mathbb{R}^p$0 such that the true dynamics are $y_t \in \mathbb{R}^p$1.

The framework employs:

• A shallow encoder $y_t \in \mathbb{R}^p$2, $y_t \in \mathbb{R}^p$3
• A recurrent latent model $y_t \in \mathbb{R}^p$4, $y_t \in \mathbb{R}^p$5
• A shallow decoder $y_t \in \mathbb{R}^p$6, $y_t \in \mathbb{R}^p$7

Superscripts $y_t \in \mathbb{R}^p$8 denote forecast and analysis, respectively.

2. SHRED Architecture and Implementation

SHRED employs an encoder-decoder sequence without a traditional autoencoder inverse. The encoder $y_t \in \mathbb{R}^p$9 is either a single linear layer or a small MLP mapping full-state snapshots into a low-dimensional latent space. The decoder $N$0 is shallow, typically a single linear layer (possibly with a nonlinearity), that reconstructs the full grid from latent codes.

Temporal dynamics in latent space are captured via $N$1, usually instantiated as an LSTM or small RNN:

$N$2

For simulation-only training, reconstruction is enforced via:

• $N$3
• $N$4
• $N$5

with mean-square error minimization over simulated trajectory $N$6.

3. Latent Data Assimilation Procedure

At each time step, the procedure executes:

• Forecast: $N$7
• Innovation: $N$8
• Analysis update: $N$9, with $\,\dot{x} = N(x, t)\,$0 as the gain matrix mapping innovations to latent corrections.

Post-update, full-state is decoded: $\,\dot{x} = N(x, t)\,$1, supporting comparisons in sensor or full-domain space.

4. Discrepancy Modeling via Sparse Identification

DA-SHRED includes a sparse regression stage to model missing physics in latent space using SINDy (Sparse Identification of Nonlinear Dynamics). For an assimilated latent trajectory $\,\dot{x} = N(x, t)\,$2, finite-difference approximations yield $\,\dot{x} = N(x, t)\,$3.

Missing latent dynamics are hypothesized to be sparse in a dictionary $\,\dot{x} = N(x, t)\,$4 of candidate nonlinear functions. SINDy regression solves:

$\,\dot{x} = N(x, t)\,$5

where $\,\dot{x} = N(x, t)\,$6, $\,\dot{x} = N(x, t)\,$7, and nonzero entries of $\,\dot{x} = N(x, t)\,$8 identify active nonlinearities. Physical corrections $\,\dot{x} = N(x, t)\,$9 are projected back to physical space via the decoder basis.

5. Training Objectives and Joint Optimization

The overall learning problem jointly tunes:

• Encoder-decoder parameters $\,y_t = H x_t + \eta_t\,$0
• Latent recurrent model $\,y_t = H x_t + \eta_t\,$1
• Assimilation gains $\,y_t = H x_t + \eta_t\,$2
• SINDy coefficients $\,y_t = H x_t + \eta_t\,$3

The main loss components are:

1. Simulation-only reconstruction:

$\,y_t = H x_t + \eta_t\,$4

2. Data-assimilation loss:

$\,y_t = H x_t + \eta_t\,$5

3. Discrepancy (SINDy) loss:

$\,y_t = H x_t + \eta_t\,$6

Combined optimization:

$\,y_t = H x_t + \eta_t\,$7

with $\,y_t = H x_t + \eta_t\,$8 as weighting hyperparameters.

6. Representative Test Cases and Quantitative Evaluation

Empirical evaluations cover:

• 2D damped Kuramoto–Sivashinsky (KS) system on $\,y_t = H x_t + \eta_t\,$9
• 2D Kolmogorov flow (Navier–Stokes with sinusoidal forcing)
• 2D Gray–Scott reaction–diffusion system
• 1D rotating detonation engine (RDE) model

Metrics include full-field RMSE, $H \in \mathbb{R}^{p \times n}$0, and sensor RMSE, $H \in \mathbb{R}^{p \times n}$1.

Key outcomes:

• DA-SHRED achieves %%%%52$y_t \in \mathbb{R}^p$053%%%% reduction in full-field RMSE within $H \in \mathbb{R}^{p \times n}$4–$H \in \mathbb{R}^{p \times n}$5 time units, compared to the simulation-only proxy.
• Robust correction with few sensors: $H \in \mathbb{R}^{p \times n}$6 simulated, $H \in \mathbb{R}^{p \times n}$7–$H \in \mathbb{R}^{p \times n}$8 real.
• SINDy module precisely recovers missing dynamical terms, e.g., $H \in \mathbb{R}^{p \times n}$9 in KS, $\eta_t$0 in Kolmogorov flow, $\eta_t$1 in Gray–Scott, $\eta_t$2 in RDE.

7. Synthesis, Practical Implications, and Extensions

DA-SHRED unites three major components:

1. Efficient compression of high-dimensional PDE states via a shallow encoder–recurrent–decoder structure yielding a compact latent representation amenable to rapid computation.
2. Latent assimilation loop implementing Kalman-style updates for incorporating sparse, noisy sensor data in real time.
3. Physics-informed discrepancy inference through sparse regression (SINDy) in latent coordinates, facilitating explicit identification of missing or uncaptured processes.

This synergy supports robust closure of the SIM2REAL gap—empirically %%%%63$y_t \in \mathbb{R}^p$064%%%% RMSE reduction compared with pure simulation—and enables interpretable extraction of dynamical corrections (Bao et al., 1 Dec 2025). The approach generalizes to a variety of physical systems and sensor modalities, providing a scalable, computationally efficient framework for digital-twin deployment, model correction, and high-fidelity state reconstruction.