Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications

Abstract: Sequence learning is dominated by Transformers and parallelizable recurrent neural networks (RNNs) such as state-space models, yet learning long-term dependencies remains challenging, and state-of-the-art designs trade power consumption for performance. The Bistable Memory Recurrent Unit (BMRU) was introduced to enable hardware-software co-design of ultra-low power RNNs: quantized states with hysteresis provide persistent memory while mapping directly to analog primitives. However, BMRU performance lags behind parallelizable RNNs on complex sequential tasks. In this paper, we identify gradient blocking during state updates as a key limitation and propose a cumulative update formulation that restores gradient flow while preserving persistent memory, creating skip-connections through time. This leads to the Cumulative Memory Recurrent Unit (CMRU) and its relaxed variant, the $α$CMRU. Experiments show that the cumulative formulation dramatically improves convergence stability and reduces initialization sensitivity. The CMRU and $α$CMRU match or outperform Linear Recurrent Units (LRUs) and minimal Gated Recurrent Units (minGRUs) across diverse benchmarks at small model sizes, with particular advantages on tasks requiring discrete long-range retention, while the CMRU retains quantized states, persistent memory, and noise-resilient dynamics essential for analog implementation.

• The paper proposes CMRU and its αCMRU variant that mitigate gradient blocking by using a cumulative update mechanism, enabling robust learning in ultra-low power settings.
• Experiments on benchmarks like sMNIST and long-sequencing tasks show that CMRU achieves high accuracy and stability, far outperforming traditional BMRU, LRU, and minGRU models.
• By controlling the ε parameter, the architecture allows precise state retention and flexible dynamics, offering promising hybrid solutions for digital and analog sequence models.

Improving the Performance and Stability of Parallelizable RNNs for Ultra-Low Power Applications

Introduction and Motivation

The drive toward deploying sequence models in resource-limited contexts (e.g., always-on sensors, biomedical implants) has underscored two central challenges: minimizing power consumption and ensuring effective long-range dependency modeling. Transformer architectures, while dominant in performance, incur prohibitive complexity for long sequences due to their quadratic scaling and hardware unfriendliness. Linear, parallelizable RNNs—particularly state-space models (SSMs) and variants such as the LRU and minGRU—offer more efficient alternatives, leveraging associative scans and linear state transitions to enable scalable, low-latency, and parallel training. However, these models inherently possess "fading memory," where information decays exponentially with time, limiting their capacity for indefinite retention, a trait essential for persistent-memory tasks and robust analog deployment.

The Bistable Memory Recurrent Unit (BMRU) was previously introduced to specifically address persistent memory in ultra-low power analog settings via quantized, bistable dynamics that map directly to Schmitt triggers. The limitation of BMRU lies in poor convergence behavior and limited expressivity—chiefly due to gradient blocking in update mode, which impedes effective learning of nontrivial sequential computations. This paper proposes the Cumulative Memory Recurrent Unit (CMRU) and its continuous-valued variant ($\alpha$CMRU) as novel, parallelizable RNN cells that overcome these limitations while retaining the hardware efficiency and persistent memory properties required for ultra-low power operation.

Cumulative Memory Recurrent Unit: Architecture and Theoretical Foundations

The core insight initiating the CMRU is the identification of gradient blocking as the limiting factor in BMRU training. In BMRU, when the state gating variable $z_t = 1$, the previous hidden state is replaced, annihilating the gradient path and severely hampering credit assignment across long chains of updates. The CMRU introduces a cumulative update controlled by a hyperparameter $\varepsilon \in [0,1]$, interpolating between full reset ($\varepsilon=0$) and perfect accumulation ($\varepsilon=1$).

$$h_t = z_t \cdot (\hat{h}_t \cdot \alpha + \varepsilon h_{t-1}) + (1-z_t) \cdot h_{t-1}$$

where $z_t$ is the (binarized) event-gating variable and $\hat{h}_t$ is the candidate update. For $\varepsilon=1$, the cell acts as a selective, quantized integrator, permitting robust gradient propagation in both update and retain modes, with eigenvalues exactly at unity. For $0 < \varepsilon < 1$, exponential forgetting is driven not by elapsed time, but by the sequence of update events—facilitating event-driven, robust state preservation.

The fixed point structure is fundamentally altered relative to BMRU, which admits only $z_t = 1$0 discrete states. At $z_t = 1$1 the reachable fixed points expand to the full lattice $z_t = 1$2, enabling both persistent and cumulative memory (Figure 1).

Figure 1: Fixed point structure of the hidden state for CMRU as a function of $z_t = 1$3: discrete bistable points at $z_t = 1$4, expanding to a lattice for $z_t = 1$5.

The $z_t = 1$6CMRU further relaxes quantization by making $z_t = 1$7 input-dependent, yielding dense, continuous-valued latent states, which theoretically approach the expressivity of digital RNNs while retaining the favorable hardware traits of CMRU when applied to analog analogs.

Empirical Evaluation: Convergence, Robustness, and Benchmark Performance

Gradient Flow and Convergence: Systematic experiments on sMNIST reveal that increasing $z_t = 1$8 yields dramatic improvements in both peak accuracy and convergence robustness. CMRU with $z_t = 1$9 achieves 96% mean accuracy with minimal seed sensitivity, while for $\varepsilon \in [0,1]$0 (BMRU), average accuracy stagnates at ~30% (Figure 2). Notably, $\varepsilon \in [0,1]$1CMRU matches or outperforms LRU and minGRU, demonstrating that cumulative updates, rather than quantization flexibility, are the primary factor in robust learning.

Figure 2: sMNIST accuracy versus $\varepsilon \in [0,1]$2, and scaling with state dimension. Cumulative updates dramatically improve both trainability and final accuracy.

Scaling and Initialization Sensitivity: Over a range of state dimensions, both CMRU and $\varepsilon \in [0,1]$3CMRU exhibit robust, low-variance convergence, especially in the small-capacity regime crucial for analog deployment, significantly surpassing LRU and minGRU in both mean accuracy and cross-seed variance.

Persistent Memory and Long-Sequencing: On copy-first tasks (both continuous and discrete settings), CMRU with $\varepsilon \in [0,1]$4 achieves near-theoretical optimal memory encoding, maintaining performance as sequence length increases to 10,000. Comparable LRU and minGRU architectures fail entirely in this regime, manifesting both high MSE/MAE and high cross-seed variability, reflecting their inherent fading memory limitations and absence of true retention dynamics (Figure 3).

Figure 3: Accuracy on copy-first-input tasks as a function of sequence length; CMRU and $\varepsilon \in [0,1]$5CMRU maintain perfect retention, while LRU and minGRU degrade precipitously.

General Sequential Benchmarks: In classical benchmarks (sCIFAR10, ListOps, IMDb, Pathfinder), CMRU and $\varepsilon \in [0,1]$6CMRU are competitive with LRU and minGRU. On tasks requiring compositional discrete state (e.g., ListOps, Pathfinder), CMRU decisively outperforms fading-memory baselines at minimal capacity, highlighting the architectural specialization: fading-memory excels at smooth integration, while persistent-memory architectures dominate tasks requiring indefinite retention and precise symbolic state.

Figure 4: The common backbone architecture: interleaved recurrent and MLP layers, with varied recurrent cells (CMRU, $\varepsilon \in [0,1]$7CMRU, LRU, minGRU).

Analog and Digital Hardware Implications: Single-layer CMRU with $\varepsilon \in [0,1]$8 achieves $\varepsilon \in [0,1]$995% accuracy on keyword spotting benchmarks, validating practical deployment at sub-microwatt power in deployed analog systems. An explicit analog circuit is proposed, mapping the cumulative update directly to cascaded BMRU cells with analog integrators (Figure 5).

Figure 5: Conceptual diagram of the native CMRU analog circuit: cascaded BMRU stages with charge integration enable cumulative fixed-point storage in hardware.

Reflection, Expressivity, and Theoretical Implications

A notable limitation emerges for tasks requiring modular or reflection dynamics (e.g., parity). Standard CMRU or $\varepsilon=0$0CMRU (with $\varepsilon=0$1) fails on parity, as lattice dynamics cannot encode periodic modular relations. However, setting $\varepsilon=0$2 induces state reflection at updates, yielding negative eigenvalues and perfect periodic alternation—aligning with recent theoretical results showing negative-real-part eigenvalues are necessary and sufficient for state-tracking tasks in linear RNNs. This highlights that control over the state transition spectrum, enabled by parameterizable $\varepsilon=0$3, is essential for expanding parallelizable RNN expressivity beyond traditional state-space architectures.

Practical and Theoretical Implications

Ultra-Low Power Deployment: The CMRU provides a scalable, robust, trainable primitive for persistent memory in analog neural hardware. Training with $\varepsilon=0$4-annealing allows models to be optimized with maximal gradient flow and then mapped to existing Schmitt-trigger hardware operating with strict bistable dynamics for actual deployment, enabling operation far below milliwatt power levels.

Hybrid Sequence Models: The complementarity observed between persistent-memory and fading-memory cells (e.g., CMRU/LRU or minGRU hybrids) suggests new architectural directions: hybrid stacks combine perfect retention with smooth integration, optimizing both learning stability and task generalization. Character-level language modeling reveals that these hybrid networks slightly outperform either family alone, and reinforcement learning literature supports the critical role of persistent memory for horizon generalization.

Input-Dependent Spectral Control: The direct relationship between $\varepsilon=0$5 and the eigenvalues of the SSM state transition matrix in CMRU generalizes the architecture: learning or adapting $\varepsilon=0$6 across layers or tasks may allow models to trivially interpolate between fading, persistent, and reflection-based computation, subsuming classic linear RNN theory within a unified, input-dependent framework.

Conclusion

The development of the CMRU and $\varepsilon=0$7CMRU constitutes a substantial advancement in ultra-low power, parallelizable RNN design. By resolving gradient blocking via cumulative updates, these cells combine persistent memory, robustness to initialization, and minimal task-specific hyperparameter sensitivity, all while maintaining practical deployment in ultra-low power hardware. CMRU stands as the first trainable, persistent-memory RNN cell suitable for analog, hardware-software co-design and immediate deployment. Echoing the authors' empirical findings, the specialization of fading- and persistent-memory architectures provides clear guidance for future hybrid systems in both digital and analog ML—where persistent, robust state retention is necessary for horizon generalization and real-world, energy-constrained deployment.

---

Figures referenced:

• Figure 1: Fixed point expansion via cumulative updates.
• Figure 2: Accuracy as a function of $\varepsilon=0$8 and scaling in sMNIST.
• Figure 5: Conceptual and transistor-level analog CMRU architecture.
• Figure 4: Experimental common backbone schematic.
• Figure 3: Copy-first-input task performance.