Decoupled Mixed-Precision Memory Hierarchy

• Decoupled mixed-precision memory hierarchies are architectures that separate high-throughput, low-precision computation from high-precision error correction to optimize performance and energy efficiency.
• They employ a dual-tier system where low-precision units (e.g., PCM crossbars, quantized caches) perform bulk arithmetic, while dedicated high-precision units handle residuals, corrections, and control tasks.
• This design underpins scalable solutions in large-scale linear algebra, deep learning, and inference on specialized hardware, significantly reducing energy consumption and alleviating memory bottlenecks.

A decoupled mixed-precision memory hierarchy is a system architecture that physically and logically separates memory and compute units, leveraging a stratified memory organization in which tasks are divided or “decoupled” between high-throughput, low-precision computation elements and high-precision, numerically robust logic or storage. This approach is foundational for energy and memory efficient execution of large-scale linear algebra, deep learning, and inference workloads, especially as device-level nonidealities, memory bandwidth, and power become limiting factors. Such hierarchies have been demonstrated in in-memory computation with phase-change memory (PCM) crossbars for scientific computing and machine learning (Gallo et al., 2017, R. et al., 2017, Nandakumar et al., 2020), in modern LLM inference using multi-level caches and dynamic quantization (Peng et al., 2024, Tang et al., 2024), and on hardware accelerators such as the Huawei Ascend 910 NPU (He et al., 23 Jan 2026). The core enabler is the assignment of memory and compute responsibilities across orthogonal precision and location domains, such that the numerical sensitivity of a computation is handled where precision is most tractable, and data movement and throughput are optimized by massive parallelism and quantization at the memory-compute boundary.

1. Fundamental Principles of Decoupled Mixed-Precision Memory Hierarchies

Decoupled mixed-precision memory hierarchies originate from the realization that although lower-precision computation (e.g., 4–8 bit quantization, analog accumulation) is orders of magnitude more area- and energy-efficient than high-precision digital logic, it cannot natively support the statistical robustness and numerical dynamic range needed for convergence and accuracy in scientific and machine learning settings. By decoupling—i.e., isolating—the high-throughput bulk arithmetic from numerically sensitive or infrequent operations, these architectures allocate:

• Low-precision, high-throughput tier: Memristive crossbars, scratchpad arrays, or quantized GPU/TPU memory modules perform matrix-vector (or tensor) arithmetic at limited precision, utilizing analog physics (Ohm’s/Kirchhoff’s laws) or packed integer formats (e.g., INT4) (Gallo et al., 2017, He et al., 23 Jan 2026). Data transfers into these arrays and out are quantized (e.g., 8-bit ADC/DAC), and effective precision is device-, quantizer-, or bandwidth-limited.
• High-precision, low-throughput tier: Von Neumann CPUs, digital gradient accumulators, or hierarchical caches hold control data, critical loop variables, or infrequently updated accumulator state in FP32/FP64 precision. Correction, residual computation, convergence checks, and precise weight updates are handled here (Nandakumar et al., 2020, R. et al., 2017, Peng et al., 2024).
• Decoupled workflow: Core computational tasks are mapped to the low-precision tier, while error correction, iterative refinement, gradient accumulation, and variable “writebacks” are handled in the high-precision tier, “decoupling” physical data layout and logical compute responsibilities.

This paradigm results in architectures where physical data location, memory transfer scheduling, and precision assignment are strategically orthogonal, allowing workload-driven adaptability.

2. Representative Architectures and Block Diagrams

In-Memory and Crossbar-Based Architectures

Le Gallo et al. (2018) and related works formalize a hybrid pipeline (Gallo et al., 2017, Nandakumar et al., 2020, R. et al., 2017):

• High-Precision Processing Unit (HPU): Standard CPU/GPU. Handles (a) outer-loop control, (b) high-precision variable storage (64-bit), (c) residual computations, preconditioning, vector updates, and (d) backward error correction (iterative refinement).
• Computational Memory Unit (CMU): 2D N×M crossbar of PCM or other resistive memory. Matrix stored as PCM conductances, physical voltages encode vectors, analog summing yields matrix-vector products at 4–6 effective bits. Device programming performed as infrequent pulses, triggered by digital “χ-buffer” overflow.
• Auxiliary Precision Buffer (“χ-buffer”): Full-precision RAM/SRAM storing accumulated updates to each matrix element until their magnitude exceeds the reliable PCM granularity ε (for weight update).
• Interconnect and Control: Digital-analog conversions (ADC/DAC), double-buffered memory transfer, drift calibration, program-and-verify device routines.

Diagram (abstracted):

[CPU/HPU] <-> [ADC/DAC, Routing, Program Control] <-> [PCM Crossbar/CMU] <-> [ADC/DAC] <-> [CPU/HPU] ↑ ↑ (High-precision residuals, (Low-precision analog MVMs, updates, control flow) stored matrix conductance)

Multi-Level Cache Architectures for LLM Inference

For large LLMs unable to fit in GPU HBM, hierarchical mixed-precision cache schemes are realized as in M2Cache (Peng et al., 2024) and HOBBIT (Tang et al., 2024):

• L1: HBM (High Bandwidth Memory): Mixed-precision, neuron-level or expert-level cache. Most critical weights are promoted to higher precision within the HBM-resident buffer.
• L2: DRAM: Layer-aware, coarse-grained cache for intermediate-sized model partitions not fitting in HBM.
• L3: SSD: Full-precision, complete model storage, accessed asynchronously for non-resident layers/experts.
• Precision assignment and replacement: Importance-ranked, dynamic per-neuron/expert assignment (FP16, INT8, INT4, etc.), with miss/hit management determined by access patterns, usage frequency, and workload constraints.

Domain Specific Accelerators: Decoupled Cores and Memory

For hardware such as the Ascend 910 NPU (He et al., 23 Jan 2026):

• Vector cores: SIMD pipelines perform on-the-fly dequantization from INT4 (packed) to FP16 (matrix tiles), fetching raw data from global memory.
• Cube cores: Dedicated high-throughput units perform FP16×FP16→FP16 MMAD (matrix multiply-accumulate), accumulating results.
• Decoupled scratchpad hierarchy: All communication between vector and cube cores mediated through global memory (no on-chip write), with programmable software-managed buffers (L0A/L0B/L0C).
• Splitting and synchronization: Divide compute into tiles/splits for concurrency and synchronize via explicit memory transfers.

3. Dataflow and Algorithmic Patterns

A unifying element across all decoupled mixed-precision hierarchies is the dataflow separation, ensuring that memory, arithmetic, and control actions can proceed asynchronously or orthogonally.

• Iterative refinement for numerically robust solvers (Gallo et al., 2017):

1. Compute a high-precision residual $r^{(k)} = b - A x^{(k)}$.
2. Solve $A \Delta x^{(k)} = r^{(k)}$ approximately using low-precision (PCM) hardware (inner CG/GMRES iteration on in-situ matrix).
3. Update solution $x^{(k+1)} = x^{(k)} + \Delta x^{(k)}$ in high-precision unit.
4. Iterate until convergence in outer high-precision loop.

• High-precision buffer “writeback” for DNN training (Nandakumar et al., 2020, R. et al., 2017):
  • Digital tier accumulates $\Delta W_{ji}$ (gradient) in a buffer $\chi_{ji}$ until $|\chi_{ji}|\geq \varepsilon$, then applies a batch analog update and resets $\chi_{ji}$ by subtracting the quantized increment.
  • Sparse device updates, compensation for device stochasticity and drift—high precision is preserved in $\chi_{ji}$, not in crossbar state.
• Multi-level cache management for LLMs (Peng et al., 2024, Tang et al., 2024):
  • At each inference step, select the most “important” neurons/experts using per-token or per-sequence criteria.
  • Assign and cache relevant modular weights at the highest available precision/bandwidth, evicting or demoting less critical blocks.
  • Dynamically move (evict/promote) modular units between HBM/DRAM/SSD as dictated by workload.

Key precision assignments:

Function	Tier/Unit	Typical Precision
MVM (in-memory, crossbar)	Analog/PCM	4–6 bits (noise-limited)
Gradient accumulation	Digital/SRAM	32–64 bits (FP/INT)
Expert/neuron cache (HBM)	HBM/DRAM/SSD	Mixed: FP16/INT8/INT4
Convergence/residual checks	CPU/GPU	FP64 or FP32

4. Device-Level Considerations, Compensation, and Bottlenecks

In-Memory and Resistive Devices

• Variability: PCM arrays suffer inter-device variability (scatter in programmed conductance), intra-device drift (conductance changes over time), and low-frequency noise.
• Compensation: Use $K$-device averaging (reduces noise as $1/\sqrt{K}$), periodic drift calibration, program-and-verify cycles, and digital correction for I–V nonlinearity (Gallo et al., 2017).
• Programming efficiency: Blind, non-iterative updates on χ-buffer overflow avoid high-frequency, low-value writes; sparse updates dominate.

Accelerator Core Decoupling

• Bottleneck origin: On Ascend 910, vector and cube cores communicate only through global memory. On-the-fly dequantization incurs extra memory traffic: vector core writes dequantized FP16 tiles to GM, cube core reads them back, incurring a 3X overhead vs. direct FP16 loading (He et al., 23 Jan 2026).
• Performance impact: Dequant compute itself only accounts for 8–12% of cycles (fully overlapped with double-buffered memory load/store). Extra traffic for INT4→FP16 round-trips is the performance limiter: maximal realized speedup from W4A16 is 1.48× compared to the 4× theoretical.
• Design recommendations: Direct scratchpad-to-scratchpad or in-MTE dequantization would close the decoupling gap, enabling as much as a 4× memory reduction to be realized in latency terms.

5. Applications, Performance, and Scalability

Scientific Computing and Deep Learning

• Solving large linear systems (Gallo et al., 2017): Systems of up to N=5000 equations solved with machine precision (error $\approx 10^{-15}$) using only 23 outer loops for N=5000, compared to $\sim$50 for fully digital CG, using 998,752 PCM devices.
• Deep learning (DNN training/inference) (Nandakumar et al., 2020, R. et al., 2017):
  • MNIST 2-layer perceptron: $97.73\%$ accuracy (vs. $98.3\%$ software FP32 baseline).
  • Sparse PCM updates $>$1000$\times$ reduction.
  • Energy savings up to $173\times$ (in mixed-precision vs. FP32 digital ASIC), and $271\times$ for matrix-vector operations.
• LLM inference on memory-constrained hardware (Peng et al., 2024, Tang et al., 2024):
  • LLaMA-70B using 24GB HBM with multi-level cache (M2Cache): 10.5$\times$ throughput, 7.7$\times$ lower carbon, 50–70% HBM reduction vs. Zero-Infinity (Peng et al., 2024).
  • MoE LLMs on Jetson AGX Orin/RTX4090 with HOBBIT: up to 13$\times$ decoding speedup, 40–55% GPU memory reduction, $<$1% accuracy drop, with per-expert adaptive precision assignment (Tang et al., 2024).

Scalability and Extensions

• Crossbar scaling: Larger or multiplexed crossbars, preconditioning, and error-correction scale the approach to higher bandwidths and model sizes.
• Task generality: Approach extends beyond matrix solve/train—applies to sparse solvers, CNNs, LSTMs, GANs, logistic regression, and general iterative optimization (Gallo et al., 2017, Nandakumar et al., 2020, R. et al., 2017).
• Accelerator support: Future decoupled hierarchies can fuse dequantization with matrix multiply, support direct low-precision paths, and enable hardware crossbars between functional units for maximal throughput.

6. Impact, Limitations, and Future Directions

Decoupled mixed-precision memory hierarchies fundamentally shape the possible efficiency/accuracy tradeoff curve as device and memory scaling becomes a primary bottleneck. Their deployment enables:

• Machine-precision numerical results with sub-8-bit device components, via robust outer-loop correction (Gallo et al., 2017).
• Convergent deep learning even with stochastic, drift-prone weight storage (Nandakumar et al., 2020, R. et al., 2017).
• Practical, sustainable large-model inference (LLMs) on legacy and cost-constrained hardware, reducing memory and energy by multiples and enabling democratized model access (Peng et al., 2024, Tang et al., 2024).
• On modern NPUs, the main bottleneck is not the local mixed-precision compute but required global memory movement due to lack of an on-chip, cross-core path for quantized data (He et al., 23 Jan 2026).

A plausible implication is that future hardware and algorithmic designs will further pursue direct, low-latency, mixed-precision data paths, more granular runtime precision assignment (not just per-weight but per-activation, per-neuron, per-token), and tighter coupling of software scheduling with physical memory/compute orchestration.

---

Selected References:

• "Mixed-Precision In-Memory Computing" (Gallo et al., 2017)
• "Mixed-precision training of deep neural networks using computational memory" (R. et al., 2017)
• "Mixed-precision deep learning based on computational memory" (Nandakumar et al., 2020)
• "Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching" (Peng et al., 2024)
• "HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference" (Tang et al., 2024)
• "W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUs" (He et al., 23 Jan 2026)