Finite-Lag Operator Geometry of Recurrent Representations

Abstract: Recurrent representations are trajectories, but representation geometry is often measured from static snapshots. We develop finite-lag operator geometry for recurrent hidden states from observed source-successor pairs $(X_t,X_{t+Δ})$. The primitive is the conditional transport law $Q_Δ(dy\mid x)$, estimated by a dense Gaussian source-smoothing operator. From this directed finite-lag law we derive a source-centered transport tensor $G_Δ$, which decomposes exactly into conditional spread and coherent displacement, and an antisymmetric coordinate circulation $W_Δ^ρ$, which summarizes directed lagged flow. We prove affine covariance with explicit metric dependence of scalar summaries, dense estimator stability on bounded trajectory clouds, and a finite-lag separation result showing that source-centered transport detects deterministic recurrent motion not recorded by infinitesimal carre-du-champ geometry. A linear-Gaussian closed form calibrates the quantities in terms of the update $A_Δ$, source covariance, and innovation covariance. Controlled experiments validate the decomposition, circulation, covariance, and stability predictions. In performance matched repeat-copy networks, the framework reveals architecture dependent differences in total transport scale and coherent displacement trace, while coherent displacement fraction is metric and resolution dependent.

• The paper introduces an operator-based framework that quantifies finite-lag transport, spread, and displacement in RNN hidden states.
• It rigorously decomposes state evolution into deterministic drift and conditional spread, validated through analytic closed forms and empirical tests.
• Comparative experiments reveal architecture-specific dynamics, highlighting distinct patterns in Elman, GRU, and LSTM recurrent networks.

Finite-Lag Operator Geometry in Recurrent Neural Representations

Introduction and Motivation

The geometric analysis of neural representations has traditionally focused on static or feedforward architectures, with major tools such as spectral methods, diffusion maps, and local similarity graphs applied to clouds of feature vectors. However, recurrent neural networks (RNNs) fundamentally operate on hidden state trajectories, in which the evolution of the state—not just its static configuration—carries computational meaning. "Finite-Lag Operator Geometry of Recurrent Representations" (2607.01746) establishes a rigorous and general framework for studying the geometry of recurrent neural hidden states through operators derived from finite-lag transition laws on source-successor pairs. This framework systematically generalizes operator geometry techniques from static settings while rectifying their inapplicability to time-ordered and non-Markovian recurrence.

Theoretical Framework

The centerpiece of this work is the definition of a finite-lag conditional transport law $Q_\Delta(dy|x)$, representing the empirical law of hidden states $X_{t+\Delta}$ given $X_t = x$, without assuming the Markov property. This construction supports a set of geometric observables:

• Source-centered transport tensor $G_\Delta(x)$, quantifying the scale and direction of finite-lag hidden-state motion, with a nontrivial decomposition:

$$2G_\Delta(x) = C_\Delta(x) + \tau m_\Delta(x)m_\Delta(x)^\top$$

where $C_\Delta(x)$ is the conditional spread and $m_\Delta(x)$ is the mean displacement per lag.

• Coordinate circulation $\mathcal{W}_\Delta^\rho$, an antisymmetric cross-moment capturing non-reversible, directed flow in representation space.

These observables are estimated empirically using a dense Gaussian source-smoothing operator, which avoids the discontinuities and instability inherent in hard k-nearest-neighbor kernel estimators.

Figure 1: Controlled calibration and stability of the finite-lag observables. (A) Decomposition of transport into spread and displacement as update scale $\beta$ varies. (B,C) Monotonic scaling of components with hyperparameters. (D) Empirical Lipschitz stability of the estimators under data perturbation.

Crucially, the source-centered approach captures deterministic drift at a chosen lag—information that vanishes in infinitesimal second-order geometry (carré du champ). This separation is both mathematically proven and empirically validated. The framework does not require hidden state Markovicity or local linearization and explicitly distinguishes between intrinsic geometric structure and aspects that depend on metric or kernel resolution choices.

Structural Results

Four structural properties of the finite-lag geometric observables are rigorously established:

1. Affine Covariance: Tensorial observables are equivariant under affine reparameterizations, but scalar summaries (e.g., traces, norms) depend on the chosen metric, necessitating careful normalization or explicit reporting of the metric used.
2. Density Estimator Lipschitz Stability: The dense Gaussian estimator is shown to be Lipschitz-stable on bounded sets at any fixed bandwidth, in stark contrast to hard k-NN graphs which can change discontinuously under infinitesimal perturbations.
3. Finite-Lag Separation Theorem: Coherent, deterministic motion (e.g., cyclic or shift updates) produces measurable transport at finite lag even when infinitesimal generator-based geometry is strictly zero.
4. Linear-Gaussian Closed Form: For recurrent updates of the form $X_{t+\Delta} = A_\Delta X_t + \xi_t$, explicit closed-form expressions for all observables are derived, providing analytic calibration akin to the static Gaussian bridge for feedforward representations.

Empirical Validation and Architecture Study

The controlled experiments validate the theoretical predictions:

• Population-sampled linear-Gaussian dynamics verify the decomposition and scaling of transport, spread, and displacement—maximum relative errors are below 0.7%.
• The empirical Frobenius norm of coordinate circulation follows the predicted linear relationship with rotation parameter, vanishing for isotropic contraction and peaking for oriented cyclic shifts.
• Stability tests confirm the observed near-linear scaling of observables with perturbation magnitude.

  Figure 2: Controlled circulation sweep. The Frobenius norm of coordinate circulation increases with the oriented rotation parameter $X_{t+\Delta}$0.

In the repeat-copy RNN case study, Elman, GRU, and LSTM architectures are compared under tightly matched performance and parameter count regimes. The main findings include:

• Total finite-lag transport scale (trace of $X_{t+\Delta}$1) and coherent displacement trace show robust, architecture-dependent separation: Elman networks consistently exhibit 15% higher transport and up to 26% higher displacement than their gated counterparts under identical task and normalization configurations.
• Coherent displacement fraction and circulation metrics are found to be highly sensitive to bandwidth, normalization, and metric choice. They are therefore less robust for comparative claims between architectures but are informative for within-architecture and bandwidth-sweep analyses.

Phase-resolved analysis in repeat-copy illuminates when (in the execution cycle) recurrent states move most. Elman networks concentrate displacement during write and recall phases, while GRU and LSTM suppress movement during memory phases, only increasing transport at recall.

Figure 3: Phase-resolved finite-lag geometry on repeat-copy with delay. Transport and coherent displacement metrics are sharply phase-localized, with each architecture exhibiting distinct dynamical regimes.

Methodological Implications and Best Practices

The framework advances the geometric analysis of recurrent representations through several methodological strengths:

• Avoids the pitfalls of hard neighborhood graphs, ensuring estimator continuity and reproducibility.
• Makes explicit the finite-lag object of measurement instead of imposing inaccessible continuous-time limits or Markov assumptions. This is particularly relevant for input-driven or complex, trained RNNs where strict Markovianity never holds.
• Justifies explicit reporting of normalization, metric, and kernel bandwidth, as these govern the interpretation and comparison of derived scalar observables.

The linear-Gaussian bridge provides a powerful analytic reference to interpret empirical results in both synthetic and trained network settings.

Implications and Future Directions

This operator-theoretic approach bridges the gap between static kernel geometry and the dynamical nature of RNNs, enabling rigorous, operator-based characterization of computation in sequence models. The demonstrated separation between total transport, spread, and displacement suggests new axes along which to compare and diagnose gated vs. ungated networks, architectural modifications, and increased model capacity.

In practical terms, the tools developed could be extended to more complex sequence models (e.g., transformers in autoregressive settings), as well as to the analysis of neural data with strong temporal structure. On the theoretical side, further generalization to non-Euclidean geometry and to continuous-time latent variable models could be developed, leveraging the operator-centric, lag-dependent formalism pioneered here.

Importantly, care must be taken in using scalar observables—absolute values can be heavily bandwidth- or metric-dependent. For precise comparisons, comprehensive reporting of estimator configuration is necessary.

Conclusion

"Finite-Lag Operator Geometry of Recurrent Representations" introduces a principled operator-based formalism for analyzing the geometry of hidden state trajectories in RNNs, capturing aspects of state evolution invisible to snapshot analyses. By decomposing the contributions of conditional spread and deterministic displacement and by introducing coordinate circulation diagnostics, it establishes a versatile toolkit for dynamical representation analysis. The methodology is supported by theoretical proofs, analytic calibration, and architecture-level empirical investigation, and should serve as a reference point for future work in geometric and dynamical understanding of learned sequence models.