Markovian Embeddings for Non-Markovian Dynamics

• Markovian embeddings are techniques that transform history-dependent processes into Markov models by extending the state with auxiliary or hidden variables.
• They employ methods such as auxiliary variables and filtrations to capture delayed dependencies, enabling standard analytical and computational Markov tools.
• Applications span quantum dynamics, anomalous diffusion, and game theory, providing tractable frameworks for simulation, inference, and control in complex systems.

A Markovian embedding is a general construction that recasts a process or system with non-Markovian (history-dependent, memory-retaining, or path-dependent) dynamics as a Markov process on an extended, typically higher-dimensional, state space. This is achieved by augmenting the system's state with additional “hidden” or “auxiliary” variables that summarize all relevant previous information, so that the future evolution depends only on the current (extended) state. The methodology is applicable in classical stochastic processes, quantum open systems, statistical mechanics, control theory, and statistical inference. Markovian embeddings enable the powerful analytical, computational, and structural tools developed for Markov processes to be applied to originally non-Markovian or time-inhomogeneous models.

1. Formal Definitions and Key Principles

Finite-State Markovian Embedding

Let $(X_t)$ be a generally non-Markovian process on a state space $\mathcal{X}$. A Markovian embedding extends the state space to $(S_t) = (X_t, \mathcal{F}_t)$, where $\mathcal{F}_t$ is a filtration (history-encoding $\sigma$-algebra) such that

$$P(S_{t+1} \mid S_t, \mathcal{F}_t) = P(S_{t+1} \mid S_t).$$

The embedded process $(S_t)$ is then Markovian (Cipolina-Kun, 2023).

Continuous-Time Embedding Problem

For a discrete-time Markov process with transition matrix $P \in \mathbb{R}^{n \times n}$, the classical embedding problem asks: does there exist a rate-matrix (generator) $Q$ with nonnegative off-diagonals and zero row sums such that

$P = e^Q~?$

If so, $P$ is the $t=1$ snapshot of a homogeneous continuous-time Markov chain (Davies, 2010, Baake et al., 2019, Casanellas et al., 2020).

Reversible Embeddings

Given a reversible $P$ (i.e., with detailed balance), the reversible embedding problem asks whether one can find a reversible $Q$ (with the same invariant distribution $\pi$) so that $P = e^Q$. Detailed results on necessary and sufficient conditions, uniqueness, and practical computation are given in (Jia, 2016, Baake et al., 27 Nov 2025).

2. Methodologies and Algorithmic Procedures

2.1 Embedding via Auxiliary Variables

For non-Markovian systems, auxiliary variables are constructed so as to encode delayed or hidden dependencies. For example, in stochastic delay differential equations,

$$\dot X(t) = F(X(t), X(t - \tau)) + \sqrt{2D_0}\,\xi(t),$$

Markovian embedding introduces a discrete-time “chain of memories” $X_1, ..., X_N$ such that the system on $(X_0, ..., X_N)$ is Markov; the limit $N \to \infty$ recovers the original delayed system (Loos et al., 2019).

2.2 Filtration-Based Embedding

In systems with path dependence (e.g., bargaining games where the future depends on past rejected proposals), extending the state with the filtration $\mathcal{F}_t$ containing entire relevant history produces a Markovian process. The transition law for the embedded process depends only on $(X_t, \mathcal{F}_t)$, not on the full path, restoring Markovianity (Cipolina-Kun, 2023).

2.3 Matrix and Spectral Characterization

For finite stochastic matrices:

• Principal Logarithm: If $P$ has positive spectrum and is reversible, the unique reversible embedding is $Q = \log(P)$, provided all off-diagonals of $Q$ are nonnegative (Jia, 2016, Baake et al., 27 Nov 2025).
• Vandermonde System: The generator in the reversible case is a polynomial in $P$ whose coefficients $(k_0, ..., k_{m-1})$ solve the linear system

$$k_0 + k_1\gamma_i + \cdots + k_{m-1}\gamma_i^{m-1} = \log \gamma_i$$

for each eigenvalue $\gamma_i$ of $P$ (Jia, 2016).

2.4 Embedding for Specialized Models

• Coupon Collection/Epidemiology: PM-matrix (upper-triangular, subset-indexed) embeddings rely on Möbius inversion to obtain explicit generator formulas and embeddability conditions (Baake et al., 2024).
• Skorokhod Topology Embeddings: Discrete-time Markov chains can be embedded into continuous time as step processes ($J_1$ embedding), linear interpolations ($M_1$), or via classical Markov (with exponentially distributed holding times); these choices determine convergence properties in weak topologies (Böttcher, 2014).

3. Applications in Physics, Probability, and Game Theory

Quantum and Stochastic Systems

• Non-Markovian quantum dynamics: Collisional models, nonlocal master equations, and delayed feedback can be embedded into a Markovian system-plus-ancilla framework. The memory kernel and waiting-time statistics are recovered either as functions of the ancilla's Lindbladian or via explicit stochastic Schrödinger equation embeddings with auxiliary wavefunctions (Budini, 2013, Li, 2020, Nurdin, 2023).
• Fractional Dynamics: Markovian embedding of fractional superdiffusion (generalized Langevin equations) is achieved by approximating the power-law kernel by a sum of exponentials and introducing auxiliary Ornstein-Uhlenbeck processes. This enables local SDE simulation of anomalous diffusion (Siegle et al., 2010).

Game Theory and Economic Networks

• Coalitional Bargaining: Markovian embeddings using filtrations enable path-dependent game dynamics (e.g., rejected proposals history) to be treated within the stochastic game/category, opening the door to Markov-perfect equilibrium analysis and standard dynamic programming (Cipolina-Kun, 2023).

4. Key Theoretical Results, Criteria, and Limitations

Table 1: Classes of Markovian Embeddings

Embedding Type	State Expansion	Embeddability Criterion
Finite-State Markov Chain	None/Auxiliary variables	$P=e^Q$ with $Q$ generator: off-diagonals $\geq0$, $\sum_j Q_{ij}=0$ (Davies, 2010, Casanellas et al., 2020)
Reversible	None	$P$ reversible, $\log(P)$ generator, unique $Q$ if exists (Jia, 2016, Baake et al., 27 Nov 2025)
Delay (SDE/Fokker-Planck)	Augment chain of $N$ memory vars	Limit $N\to\infty$ gives correct non-Markovian marginal hierarchy (Loos et al., 2019)
Quantum Ancilla Embedding	System + finite-dimensional ancilla	Lindblad QSDE for combined system, reduction via trace (Budini, 2013, Nurdin, 2023)
Filtration (Game Theory)	Augment state with filtration	$S_t = (X_t, \mathcal{F}_t)$, Markov iff $P(S_{t+1}|S_t)$ (Cipolina-Kun, 2023)

Spectral Constraints and Uniqueness

• Embeddability requires $P$ to be invertible, with eigenvalues in the “Runnenberg–Karpelevič region”—no spectrum on $(-\infty,0]$ except possible pairs of negative eigenvalues of even algebraic multiplicity (Culver’s criterion) (Davies, 2010, Baake et al., 2019, Baake et al., 27 Nov 2025).
• For reversible $P$, existence and uniqueness of a reversible embedding is guaranteed if and only if $P$ has positive spectrum and satisfies detailed balance (Jia, 2016, Baake et al., 27 Nov 2025).
• In constructive probability/statistics: under certain empirical constraints (e.g., at most one jump per interval in sampled data), a unique intensity matrix fits the observed transitions without regularization (Carette et al., 2023).

5. Computational Aspects, Algorithms, and Limitations

• Complexity: Embedding computations generally require $O(n^3)$ for eigen/singular value decompositions and Vandermonde systems (Jia, 2016).
• Practical Algorithms: The diagonal-adjusted logarithm is an optimal $\ell^\infty$-projection for non-embeddable $P$, providing the closest embeddable approximation (Davies, 2010).
• Limitation: Not all Markov chains admit a Markovian embedding. For discrete chains, embeddability is rare for $n \geq 3$ except in special models (e.g., symmetric, circulant, or special subclass matrices). Necessary conditions are spectral, but no simple spectral characterization is sufficient for $n \geq 3$ (Davies, 2010, Baake et al., 2019, Baake et al., 27 Nov 2025).
• Quantum: Dimensionality of the embedding (number of auxiliary wavefunctions or ancilla levels) depends on environmental memory spectrum; approximation error is controlled by the quality of fitting (e.g., bath spectrum) (Li, 2020, Nurdin, 2023).

6. Structural and Geometric Aspects

• Minkowski Embedding: The set of Markov generators can be isometrically embedded via the symmetrized Kullback-Leibler divergence into a space with a Minkowski metric, cleanly separating equilibrium (spacelike) and nonequilibrium (timelike) dynamical features. The geometry reveals underlying thermodynamic and information-theoretic structure of Markov dynamics (Andrieux, 2024).

7. Broader Impact and Cross-Disciplinary Connections

Markovian embedding principles unify the analysis of classical and quantum dynamics, stochastic control, inference, game theory, and statistical mechanics. They facilitate reduction of non-Markovian to Markovian models at the cost of higher-dimensional state spaces, providing tractable frameworks for simulation, control, inference, and equilibrium analysis in otherwise analytically intractable systems or path-dependent games. Embedding constructions also clarify the structural features and limitations of the underlying physical, biochemical, or economic systems.