SWE-Replay: Sequence-Wise Experience Replay

• SWE-Replay is a framework that applies sequence-wise replay of signals, transitions, and syscalls to enhance control systems, reinforcement learning, and debugging reproducibility.
• In cyber-physical systems, it augments LQG control with randomized watermarking and joint-statistics CUSUM tests to detect replay attacks up to 3x faster than traditional methods.
• For reinforcement learning and program record-and-replay, SWE-Replay uses policy-weighted sampling and hardware-assisted logging to improve sample efficiency and ensure deterministic execution with minimal overhead.

SWE-Replay refers to several distinct but conceptually allied frameworks across cyber-physical attack detection, reinforcement learning, and deterministic program execution recording. The common denominator is “Sequence-Wise Experience Replay,” an approach that leverages the replay of sequences—signals, transitions, syscalls, or control inputs—to enhance detection, learning, or reproducibility. Notable variants span joint-statistics sequential attack detectors in control systems (Naha et al., 2020), policy-weighted sample replays for RL (Sinha et al., 2020), high-utility trajectory stores for TD propagation (Karimpanal et al., 2017), and deployable execution recording systems for debugging (O'Callahan et al., 2017). This article surveys each SWE-Replay instantiation with technical rigor, placing each in the context of its domain’s foundational methods.

1. Sequential Detection of Replay Attacks in Control Systems

In cyber-physical systems, a replay attack hijacks the measurement channel by feeding past observations into the state estimator/controller, mimicking legitimate signals with near-indistinguishable statistics. The SWE-Replay method [Editor’s term] augments conventional LQG control with stochastic input watermarking and sequential detection based on joint innovation-watermark statistics (Naha et al., 2020).

The plant’s linear dynamics are described by:

$$x_{k+1} = A x_k + B u_k + w_k,\quad y_k = C x_k + v_k$$

with $w_k, v_k$ Gaussian white noise. The controller injects a random watermark $e_k \sim \mathcal{N}(0, \Sigma_e)$ so that $u_k = u_k^* + e_k$, coupling control excitation directly to the measurement innovation. Replay attacks substitute $z_k = y_{k - k_0}$, deceiving the estimator.

Detection is performed via a CUSUM test on the joint vector $[\gamma_k^T, e_{k-1}^T]^T$, comparing the likelihoods under pre- and post-attack Gaussian distributions. The test’s asymptotic delay is governed by the Kullback-Leibler divergence (KLD) between the joint distributions:

$$D(f_1\|f_0) = \frac12 \left\{ \operatorname{tr}(\Sigma_\gamma^{-1}\Sigma_{\tilde\gamma}) - m - \ln\frac{\det(\Sigma_{\tilde\gamma} - C B \Sigma_e B^T C^T)}{\det\Sigma_\gamma} \right\}$$

Detection delay scales as $$\textrm{SADD} \approx \frac{\ln(\textrm{ARL}_h)}{D(f_1\|f_0)}$$. For systems with relative degree $d_r \geq 2$, the coupling $C B$ vanishes; using delayed watermarks $e_{k-d_r}$ modifies the KLD and restores detectability.

Optimal watermark design further constrains the increase in control cost $\Delta\textrm{LQG}$, yielding significant delay reductions—Monte Carlo simulations confirm 3x improvements over batch $\chi^2$ detectors. SWE-Replay thus tightly connects statistical detection power to Markovian joint statistics and optimal randomized excitation.

2. SWE-Replay in Off-Policy Reinforcement Learning

SWE-Replay is a core technique in sample-efficient RL settings, where experience replay mechanisms are used to accelerate temporal-difference (TD) learning. Traditional methods prioritize experiences by TD error or sample randomly. SWE-Replay variants weigh sample importance according to the likelihood under the stationary policy distribution $d^\pi(s,a)$, directly targeting convergence guarantees and maximizing utility for frequently visited states (Sinha et al., 2020).

Formally, state-action pairs $(s,a)$ are sampled from the empirical buffer distribution $d^D(s,a)$; each is reweighted by $w(s,a) = \frac{d^\pi(s,a)}{d^D(s,a)}$. This ratio is approximated using a likelihood-free density-ratio estimator $w_\psi(s,a)$, trained via f-divergence minimization. The critic loss becomes

$$L_Q(\theta) = \mathbb{E}_{d^D} \left[ \tilde w(s,a) \left( Q_\theta(s,a) - [r + \gamma \, \mathbb{E}_{a'} Q_\theta(s', a')] \right)^2 \right]$$

where normalization stabilizes updates. Evaluation on MuJoCo control tasks demonstrates 10–20% sample complexity improvements over uniform and prioritized sampling. The approach is robust for deep actor-critic methods, with negligible gains in direct actor loss reweighting.

3. Sequence-Wise Transition Replay for Temporal Difference Propagation

An alternate SWE-Replay paradigm (Karimpanal et al., 2017) selects and replays entire multi-step transition sequences with high TD error, rather than isolated or randomly chosen transitions. Each sequence

$$\Theta_t = [S(x':y'),\,\pi(x':y'),\,R(x':y'),\,\Delta(x':y')]$$

records states, actions, rewards, and absolute TD errors over a window. Selection is governed by maximum TD error, with exclusivity-driven insertion and eviction from a bounded library $L$. Additionally, “virtual” sequences are constructed by splicing matching subtrajectories, allowing value function information to trickle through previously unvisited regions.

Empirically, this mechanism produces dramatic acceleration in value propagation for sparse-reward, off-policy benchmarks—e.g., puddle-world navigation and mountain-car—yielding 10–20x improvements in secondary-task returns over prioritized or uniform experience replay. The approach is adaptive to multi-task and continual learning, and enables fast correction when policy and behavior diverge.

4. Deterministic Program Record and Replay Systems

SWE-Replay also appears as a deployable record-and-replay (R&R) software framework for deterministic user-space execution (O'Callahan et al., 2017), notably in Mozilla’s rr. Instead of instrumenting code or capturing the entire VM, rr exploits hardware features of x86/Linux (ptrace, perf_events, seccomp-bpf, deterministic branch counters) to enforce single-threaded event order and intercept all syscalls:

• Syscalls: Recorded with arguments, results, and user-space memory changes. On replay, register and memory state is restored at each syscall.
• Signals: Asynchronous events mapped to deterministic hardware counters (retired conditional branches), with full register saves and synthetic replay.
• Scheduling: Single-threaded global ordering eliminates data races; context switches forced by external events or preemption are logged.
• Shared memory: Direct mapping operations from external drivers are disabled or sandboxed; only trace-mediated updates occur.

Trace files are append-only, zero-copy cloned for large mmapped files, and compressed to minimize overhead. On typical low-parallelism workloads, rr incurs ~$1.5-1.8\times$ record overhead and enables CPU-faithful replay, trading off multicore parallelism fidelity for reproducibility and deployability.

5. Structural, Theoretical, and Practical Implications

SWE-Replay frameworks share a core structural property: sequence-wise joint statistics, whether in measurement-control pairs, multi-step RL transitions, or program execution events, significantly enhance information propagation (detection, update, reproducibility) over pointwise approaches. For attack detection, the joint innovation-watermark test achieves closed-form KLD characterizations of delay and cost. In RL, policy-weighted sequence selection and virtual trajectory construction foster rapid value function adaptation under nonstationarity. R&R systems advance real-world debugging without kernel modification, leveraging deterministic event ordering rooted in hardware observables.

Limitations are domain-specific: replay attack detection can degrade in high-relative-degree systems absent delayed watermarking; RL importance weights require accurate ratio estimators and may struggle in high-dim representation domains; record-and-replay systems lose fidelity for true multicore race bugs and require platform-specific kernel support.

6. Comparative Evaluation Across Domains

A comparative overview is presented below, organizing SWE-Replay variants by domain, core mechanism, and quantitative benchmark outcome:

Domain	Mechanism	Key Metric / Result
Attack Detection	Joint CUSUM with watermark	Up to 3x faster replay detection for equal control cost
RL (TD Learning)	Likelihood-free importance replay	10–20% sample efficiency gain on control tasks
RL (TD Propagation)	Multi-step sequence + virtual	10–20x secondary-task return in sparse reward navigation
Program R&R	Syscall, signal, perf replay	$<$2x overhead for deterministic replay, CPU-level fidelity

These outcomes are directly referenced from domain-specific evaluations (Naha et al., 2020, Sinha et al., 2020, Karimpanal et al., 2017, O'Callahan et al., 2017).

7. Extensions and Open Problems

SWE-Replay mechanisms are extensible in several directions:

• RL: Integration with deep function approximation, representation learning in high-dimensional (e.g., pixel) domains, and adaptation of buffer ratios for stability.
• Control: Extension to nonlinear or uncertain dynamics, more sophisticated watermark design for higher relative-degree or constrained systems.
• Record & Replay: Support for true multicore race detection via explicit shared-memory race instrumentation, extension to non-x86 architectures contingent on hardware counter availability.
• Security: Application to networked control under coordinated or stealthy replay variants, cross-domain defense strategies leveraging joint-statistics CUSUM.

Potential cross-disciplinary implications include the transfer of joint-sequence test principles into RL, debugging, and secure control applications for improved sample efficiency, reproducibility, and resilience under adversarial conditions.