Quantifying Memory Use in Reinforcement Learning with Temporal Range

Abstract: How much does a trained RL policy actually use its past observations? We propose \emph{Temporal Range}, a model-agnostic metric that treats first-order sensitivities of multiple vector outputs across a temporal window to the input sequence as a temporal influence profile and summarizes it by the magnitude-weighted average lag. Temporal Range is computed via reverse-mode automatic differentiation from the Jacobian blocks $\partial y_s/\partial x_t\in\mathbb{R}^{c\times d}$ averaged over final timesteps $s\in{t+1,\dots,T}$ and is well-characterized in the linear setting by a small set of natural axioms. Across diagnostic and control tasks (POPGym; flicker/occlusion; Copy-$k$) and architectures (MLPs, RNNs, SSMs), Temporal Range (i) remains small in fully observed control, (ii) scales with the task's ground-truth lag in Copy-$k$, and (iii) aligns with the minimum history window required for near-optimal return as confirmed by window ablations. We also report Temporal Range for a compact Long Expressive Memory (LEM) policy trained on the task, using it as a proxy readout of task-level memory. Our axiomatic treatment draws on recent work on range measures, specialized here to temporal lag and extended to vector-valued outputs in the RL setting. Temporal Range thus offers a practical per-sequence readout of memory dependence for comparing agents and environments and for selecting the shortest sufficient context.

• The paper introduces Temporal Range that quantifies memory use in RL by leveraging Jacobian sensitivity analysis over input sequences.
• It establishes an axiomatic foundation ensuring unique, normalized metrics that robustly compare agent memory and guide context window selection.
• Empirical evaluations across control and Copy-k tasks validate that Temporal Range accurately reflects memory demands and informs deployment decisions.

Quantifying Memory Use in Reinforcement Learning with Temporal Range

Introduction

This work introduces Temporal Range, a rigorous, model-agnostic metric for quantifying the effective use of historical information by trained RL policies. The metric leverages first-order sensitivities—specifically, Jacobian blocks of vector outputs with respect to input sequences—to construct a temporal influence profile, summarized via a magnitude-weighted average lag. The approach permits fine-grained, per-sequence and per-timestep readouts of memory dependence, thereby providing a scalar diagnostic for agent and environment comparison, as well as for context window selection in deployment.

Metric Formulation and Axiomatic Foundation

Temporal Range is defined for differentiable mappings $F$, mapping observation sequences $X_{1:T}$ to vector outputs $y(X)$. For each input position $t$ and output position $s > t$, the Jacobian block $J_{s,t} = \frac{\partial y_s}{\partial x_t}$ is computed and aggregated (typically with the mean operator) over final timesteps, forming per-step influence weights $w_t$. The temporal range score is then given by:

$$\hat{\rho}_T = \frac{\sum_{t=1}^{T} w_t (T-t)}{\sum_{t=1}^{T} w_t}$$

$\hat{\rho}_T$ reflects the average look-back distance influencing the policy's decision at the final timestep.

The paper provides an axiomatic derivation guaranteeing the uniqueness of both normalized and unnormalized forms (with invariance to input/output rescaling), specialized for vector-output linear maps. In particular, single-step calibration, additivity over disjoint time indices or magnitude-weighted averaging, and absolute homogeneity uniquely fix the metric's form. This foundation ensures robust cross-agent and cross-environment comparisons.

Computational Properties and Approximation

Temporal Range is computationally efficient, requiring only the policy's differentiable outputs and standard reverse-mode autodiff machinery for Jacobian computation. When policies are non-differentiable or black-box, a proxy LEM policy can be trained to supply reliable Jacobians, leveraging LEM's stable long-horizon gradient characteristics. This proxy approach maintains fidelity against known task structure and ground-truth memory requirements.

Empirical Evaluation

The metric is validated across a suite of POPGym diagnostic environments and classic control tasks, employing architectures such as MLPs, GRUs, LSTMs, LEM, and oscillatory state-space models (LinOSS). Results confirm several claims:

• Small range in fully observed control: Tasks such as CartPole (near-Markovian) exhibit low Temporal Range, reflecting minimal memory demand.

Figure 1: CartPole task shows low effective memory demands, with policies focusing on recent timesteps.

• Scaling with task lag: In Copy-$k$ tasks, which demand recall at fixed temporal offset, measured $\hat{\rho}_T$ aligns strongly with $k$, matching ground-truth requirements as established by closed-form analysis.

Figure 2: LEM agent on Copy $k=3$ task displays a temporal influence profile peaking near the required look-back distance.

Figure 3: For Copy $k=10$, the profile peaks further back, mirroring the increased temporal memory demand.

• Alignment with window ablations: Window-ablated evaluations (rebuilding hidden state from truncated histories) reveal that performance recovers when the context size exceeds the measured Temporal Range, confirming that $\hat{\rho}_T$ accurately identifies the minimum sufficient context.

Figure 4: In CartPole, ablation studies show rapid recovery with short window sizes, consistent with low Temporal Range.

• Effect of noise and partial observability: In heavily occluded or noisy variants (e.g., Noisy Stateless CartPole), Temporal Range increases, reflecting the agent’s need for longer history integration to reconstruct state or smooth observations.

Figure 5: Noisy Stateless CartPole induces broad temporal influence profiles, indicating increased reliance on past observations.

Architectural Analysis and Deployment Implications

Temporal Range exposes material differences across architecture classes: GRUs typically yield shorter effective ranges unless forced by task structure; LEM and LinOSS often maintain extended multiscale tails indicative of slow modes; LSTMs are more sensitive to initialization. Notably, overestimation of range by certain architectures on near-Markovian tasks identifies redundant memory usage, an inefficiency revealed only by local sensitivity analysis.

The metric supports practical deployment decisions. Empirical results demonstrate that deploying large models with history windows set by measured $\hat{\rho}_T$ achieves near-baseline performance with substantial memory savings. Truncation below TR recommendations results in pronounced performance deterioration, confirming the tight coupling between measured range and necessary context.

Analytical Calibration

Closed-form computations further calibrate Temporal Range. In Copy-$k$, the normalized metric matches $k$ precisely regardless of matrix norm. For linear recurrent models, the profile and expected lag are governed by powers of the transition matrix, bridging spectral analysis and empirical range.

Implications and Future Directions

Temporal Range provides a principled tool for memory auditing in RL, agent-environment comparison, and context window selection. It enables practitioners to detect under- or over-utilization of memory, adjust architectures or deployment strategies, and identify sufficient—and not excessive—context lengths, critical for resource-constrained or embedded applications.

The measure’s locality (per rollout/timestep), dependence on preprocessing and norm choice, and susceptibility to slow mode inflation are noted limitations. Future work may involve causal ablation, regularization toward shorter range solutions, time-resolved and per-head range profiling, and extension to multi-agent and hierarchical RL.

Conclusion

Temporal Range furnishes a unique, nonparametric, and computationally direct measure of history dependence for RL agents. Its theoretical rigor, empirical reliability, and deployment value render it an effective standard for memory quantification in RL research and practice.

See (2512.06204) for complete citations.