Delay Embedding Theory in Neural Sequence Models

• Delay Embedding Theory is a framework for reconstructing a system’s latent state from time-delayed observations based on Takens’ theorem.
• Neural sequence models, including transformers and LRUs, utilize delay embeddings to efficiently capture hidden dynamics for forecasting and reconstruction.
• Empirical findings show that structured models like LRUs achieve superior embedding quality and parameter efficiency, highlighting trade-offs in noise robustness and training time.

The delay embedding theory of neural sequence models posits that the capacity of modern sequence models—such as transformers and state-space architectures—to infer unobserved or latent state from observed time series can be rigorously analyzed in terms of classical results from dynamical systems. Central to this perspective is Takens’ embedding theorem, which guarantees that, under suitable conditions, the full latent state of a dynamical system can be reconstructed from a sufficiently long history of a single observed measurement. This framework provides a quantitative, geometry-informed account for why specific neural architectures succeed or fail in reconstructing and predicting hidden variables from partial, noisy observations (Ostrow et al., 2024).

1. Theoretical Basis: Delay Embedding and Takens’ Theorem

Takens’ delay-embedding theorem states that for a compact $d$-dimensional smooth manifold $M$ and smooth diffeomorphism $\varphi: M \to M$, the history of scalar measurements $h: M \to \mathbb{R}$ forms the $m$-delay vector $$F_h^m(x) = (h(x), h(\varphi(x)), \ldots, h(\varphi^{m-1}(x))) \in \mathbb{R}^m$$. When $m \geq 2d + 1$, $F_h^m$ is generically an embedding: its image $F_h^m(M)$ is diffeomorphic to $M$. There exists an inverse (reconstruction map) $\Psi: F_h^m(M)\to M$ such that $\Psi \circ F_h^m = \text{id}_M$. Thus, stacking $m \geq 2d + 1$ consecutive time-lagged samples of an observed variable suffices to reconstruct the original system’s dynamical state.

The embedding dimension bound $m \geq 2d + 1$ guarantees non-overlapping (injective) reconstruction, but in practice $m$ may be increased for robustness to noise, and the quality of the reconstruction degrades for $m$ below the bound, especially under finite-sample or noisy conditions.

2. Connection to Neural Sequence Models

Modern sequence models process time series data as a sequence of input tokens $u_1, \ldots, u_T$ and compress them into a hidden state $h_T$. If interpreted via delay embedding, the hidden state $h_T$ serves as a learned delay coordinate embedding that unfolds the latent dynamical structure of the input. Specifically, the hidden state is trained to encode all information from a recent window $u_{T-m+1:T}$ needed both to predict future observations and reconstruct latent variables.

Empirical studies on the 3D Lorenz attractor demonstrate that even when only a noisy scalar $x(t) + \eta(t)$ is available, constructing $m$-length delay vectors with $m \approx 5$ or $6$ recovers the attractor’s geometry. Neural sequence models, provided with this partial observation, can thus be tested for their ability to learn such embeddings from data (Ostrow et al., 2024).

3. Architectural Realizations and Inductive Biases

3.1 One-Layer Transformer Decoder

A one-layer transformer decoder processes input tokens $\{u_1, \ldots, u_T\}$, applies positional encodings, and uses self-attention to compute output activations. The attention mechanism performs a learned, context-dependent selection over prior positions, implicitly choosing a sparse set of delays. The transformer’s parameter count scales as $O(d_{\text{model}}^2)$, as the weight matrices are full-rank.

3.2 Linear Recurrent Unit (LRU) State-Space Model

The LRU is a discrete-time linear state-space model governed by

$$x_{t+1} = A x_t + B u_t, \qquad o_t = C \mathrm{Re}(x_t) + D u_t,$$

where $$A \in \mathbb{C}^{d_{\text{model}} \times d_{\text{model}}}$$ is diagonal with unit-disk eigenvalues, and $B,C,D$ are appropriately sized real matrices. The LRU’s hidden state $x_t$ encodes past inputs through a superposition of complex rotations, preserving a nearly uniform and redundant delay embedding of the input history. The model’s parameter count grows as $O(d_{\text{model}})$.

3.3 Inductive Bias Comparison

At initialization, LRU’s structure inherently captures the entire past via its stable, diagonalizable $A$-matrix. This aligns with the requirement $m \geq 2d + 1$, as increasing $d_{\text{model}}$ directly increases the effective embedding window. Transformers, by contrast, start with randomly initialized attention weights and must learn to select the relevant delays. Empirically, the LRU’s hidden states exhibit coarse delay-embedding structure even before training, whereas transformer embeddings achieve similar quality only after substantial training.

4. Experimental Methodology

4.1 Synthetic Time-Series Benchmark

Experiments utilize the Lorenz attractor with parameters $\sigma=10$, $\rho=28$, $\beta=8/3$. Trajectories are generated with added Gaussian noise of variance $\sigma^2_{\text{noise}} \in \{0, 0.05, 0.1\}$ on the observed $x(t)$. A dataset of 2,000 trajectories, each of length 600 (with the initial 100 steps discarded), forms the basis for next-step prediction tasks (Ostrow et al., 2024).

4.2 Training Objective

Models are trained with Adam for 1,000 epochs on the Mean Absolute Standardized Error (MASE): $$\text{MASE} = \frac{|x_t - \hat{x}_t|}{|x_t - x_{t-1}|},$$ averaged over $t$. Only model runs with final MASE $< 1$ are analyzed.

4.3 Embedding-Quality Metrics

Evaluation includes:

• Nonlinear and linear decoding $R^2$: Decoders trained to map hidden state $o_T$ to unobserved variables $(y(T), z(T))$.
• Neighbors Overlap: Compares overlap of nearest neighbors in the true latent state and the learned hidden-state space.
• Unfolding (Conditional Variance): Assesses how well $o_t$ predicts conditional variance of future observations.
• Participation Ratio (PR): Quantifies effective embedding dimension in the hidden-state covariance.

5. Quantitative Performance and Embedding Analysis

• Parameter efficiency: For a fixed parameter count, the LRU architecture consistently achieves lower MASE than the transformer (e.g., at $d_{\text{model}} \approx 50$, LRU attains MASE $\approx 0.07$, transformer $\approx 0.12$).
• Embedding development: LRU decoders reach high decoding $R^2$ ($\approx0.7$) almost immediately, with Neighbors Overlap progressing from $\approx0.4$ to $\approx0.8$. Transformers require $\gtrsim200$ epochs to begin matching this embedding quality.
• Correlation between embedding and prediction: Strong negative correlations are observed: nonlinear decoding $R^2$ vs. MASE ($\approx -0.76$), neighbors overlap vs. MASE ($\approx -0.64$), with higher embedding quality tightly linked to better prediction accuracy.
• Noise sensitivity: Increasing observation noise ($\sigma^2_{\text{noise}}: 0 \rightarrow 0.1$) increases LRU’s MASE by $\approx40\%$, transformer by $\approx20\%$. LRU’s lower effective embedding dimension (PR $\approx 1.3$) makes it more sensitive to noise than the transformer (PR $\approx 2.5$).
• Scaling laws: No clear power-law between MASE and $d_{\text{model}}$ is observed; performance gains diminish for $d_{\text{model}} \gtrsim 50$ (i.e., values far in excess of $2d+1$).

6. Conclusions and Model Design Implications

• Inductive bias of models: State-space models such as LRUs function as near-uniform delay embedders of input history, achieving high-quality geometric embedding and parameter efficiency from initialization. Transformers, while capable of learning viable embeddings, require more parameters and training time due to the need to select informative delays from scratch.
• Embedding strength and prediction: The tight correlation between embedding quality metrics and next-step prediction accuracy confirms the utility of delay-embedding theory as an analytical framework.
• Noise and data regime considerations: In low-data or compute-constrained scenarios, structured state-space models with $m \gg 2d+1$ offer advantages. In highly noisy or nonstationary settings, transformers' selective embedding via attention may guard against overfitting measurement noise.
• Hybrid architectures: A plausible implication is the potential benefit of hybrid designs combining a state-space backbone for robust large-$m$ embeddings with attention-based mechanisms to refine context relevance.

By synthesizing classical delay-embedding theory with analysis of neural sequence models, this framework enables precise, geometry-aware criteria for evaluating and developing sequence architectures in partially observed dynamical systems tasks (Ostrow et al., 2024).