Mixed-Precision Paradigms

• Mixed-precision paradigms are computational strategies that employ multiple numerical formats to balance accuracy, speed, and resource efficiency.
• They implement methods like iterative refinement and dynamic precision assignment in routines such as linear solvers and FFT algorithms.
• These approaches boost performance in scientific computing and machine learning by reducing memory footprint and energy consumption while preserving numerical fidelity.

A mixed-precision paradigm denotes any computational strategy that combines two or more numerical formats—typically floating-point representations of differing bit widths—within the same algorithm or system to balance numerical accuracy, performance, energy efficiency, and memory usage. Rather than uniformly applying the highest available precision (e.g., double-precision everywhere), mixed-precision schemes dynamically or statically assign precision levels to different subroutines, data partitions, or hardware units, ensuring that accuracy is preserved only where it critically impacts the computation while less critical stages are executed in lower precision to leverage hardware advantages. Mixed-precision methods are now fundamental both in scientific computing and in large-scale machine learning, underpinning advances in linear algebra, numerical simulation, model reduction, optimization, and neural computation.

1. Foundational Principles and Algorithmic Structures

Mixed-precision methodologies exploit the significantly higher arithmetic throughput and lower memory requirements of low-precision formats (e.g., FP16 or FP32) by relegating the bulk of computational work to these formats while strategically invoking high-precision (e.g., FP64) computation to control rounding error or guarantee solution fidelity (0808.2794). In classical dense and sparse linear algebra, two principal algorithmic structures illustrate the paradigm:

• Iterative Refinement: Compute an initial approximate solution to a linear system using a low-precision LU or Cholesky factorization, then perform iterative corrections in high precision until the residual minimizes to within the target tolerance. Corrections can be formulated as:

$$x_{k+1} = x_k + d_{k+1}, \quad \text{with } d_{k+1} \text{ computed via low-precision triangular solves, and each residual } r = b - A x \text{ in high precision}.$$

This structure extends to inner–outer Krylov subspace schemes, such as FGMRES with GMRES$_\mathrm{SP}$ preconditioner, where the inner iterations reside in lower precision (0808.2794).

• Model Reduction and Decomposition: For matrix interpolative decomposition (ID), the most compute-intensive steps (pivoted QR and triangular solve) are performed in low precision, but the extraction of skeleton columns and final assembly is anchored on high-precision data, ensuring global decomposition accuracy (Dunton et al., 2020).
• Dynamic Stage-wise Assignment: In FFT-based algorithms, the computation is split into discrete stages (e.g., forward FFT, inverse FFT, GEMV), and for each stage, a runtime policy assigns single or double precision based on expected error contribution and measured speedups. The optimal assignment across $k$ stages minimizes runtime subject to a user-defined global error tolerance, solvable by enumerating all $2^k$ configurations (Venkat et al., 13 Aug 2025).

2. Error Analysis, Stability, and Theoretical Guarantees

Precise backward and forward error analyses have been developed to quantify under which conditions mixed-precision schemes deliver results matching high-precision implementations. Core results include:

• Solvers with Iterative Refinement: Convergence is governed by the condition number $\kappa(A)$ and the low-precision machine epsilon $\epsilon_\ell$. A sufficient condition for convergence is $\kappa(A) \cdot \epsilon_\ell < 1$ (0808.2794). For direct solvers:

$$\|b - A x_k\|_2 \leq \|A\|_2 \|x_k\|_2 \epsilon_{DP} \sqrt{n},$$

and for the relative forward error:

$$\frac{\|x_k - x_{\mathrm{true}}\|_2}{\|x_{\mathrm{true}}\|_2} = O\big(\kappa(A) \epsilon_{DP}\big).$$

• Matrix Interpolative Decomposition: The spectral norm error of a mixed-precision ID satisfies

$$\|A_D - \widehat{A}_M\|_2 \leq \sqrt{1+k(n-k)}\, \sigma_{k+1}(A_D) + C \, u_L \, \sigma_1(A_D),$$

indicating that as long as the product $u_L \sigma_1(A_D) \ll \sigma_{k+1}(A_D)$, the low-precision induced error is negligible (Dunton et al., 2020).

• Adaptive Precision in Iterative Solvers: Dynamic schemes estimate the residual gap and monitor the progress of convergence, downgrading the precision (e.g., from FP64 to FP32 or FP16) exactly when the attainable error is below a threshold, ensuring that the final residual is limited by the highest-precision arithmetic engaged (Guo et al., 7 May 2025).
• Mixed-Precision in MCMC and Sampling: In neural quantum state simulations, careful quantification of the effect of log-density perturbations on the MCMC stationary distribution yields total-variation bias scaling as $O(\sigma^2/(1-r))$, where $\sigma^2$ is the variance of precision-induced noise in $\log \pi(x)$ and $r$ is the contraction rate. This stringent bound guarantees that high-dimensional sampling is robust to aggressive down-casting for significant portions of the computation (Solinas et al., 28 Jan 2026).

3. Hardware Abstractions and Software Architectures

Contemporary mixed-precision paradigms are deeply co-designed with hardware features and runtime systems:

• Fused Multiply-Add (FMA) Units: GPU Tensor Cores (NVIDIA Volta/Turing/Ampere) support matrix computation with FP16 inputs, and accumulation in FP32, ensuring high final accuracy even with low-precision operands. The output error depends only on the accumulator precision, not on the input (Gallouédec, 2021).
• Dot-product and Matrix Engines: Modern CPU ISAs (x86_64, ARM, RISC-V) expose mixed-precision “matrix” instructions such as AMX_INT8, ARM SME2, and RISC-V IME, supporting int8/int16 inputs with int32/FP32 accumulation. Micro-kernels are tuned to make use of tile or dot-product centric layouts, shifting computation from memory-bound to compute-bound regimes and delivering 2×–8× throughput improvements over legacy designs (Martínez et al., 13 Jun 2025).
• Tile-Centric and Receiver-Side Conversion: In parallel GEMM frameworks (e.g., PaRSEC on Fugaku or A100), matrices are decomposed into tiles, and each tile is assigned a format statically or adaptively. Runtime type conversion (receiver-side) ensures hardware can always operate on the optimal data format, and PaRSEC’s parameterized DAG expresses all tile dependencies and kernel invocations. Performance scales near-linearly to thousands of nodes, with mixed-precision error tracked post-hoc by full-precision recomputation (Zhang et al., 20 Aug 2025).
• Custom Memory and Data Packing: For LLM inference, offline quantization and tiling assign bit-widths (e.g., 4-bit, 8-bit, FP16) matching hardware coalescing and alignment requirements. Attention modules handle arbitrary Q/K/V precisions by head-wise storage re-alignment and pipelined I2F conversion, saturating tensor-core throughput (e.g., 1.42–1.39× throughput speedup on Qwen 8B, 8-bit KV cache with TurboMind) (Zhang et al., 21 Aug 2025).
• In-Memory Computing: Phase-change memory (PCM) crossbars offload matrix-vector multiplies to analog devices with low-precision, non-volatile multiplication, while a digital, high-precision host processor completes residual updates and iterative refinement to guarantee final accuracy (Gallo et al., 2017).

4. Domain-Specific Applications and Performance Outcomes

Mixed-precision paradigms are ubiquitous across domains:

• Scientific Linear Algebra: Dense and sparse linear solvers, eigenproblems, and model decomposition (ID/SVD) see speedups ranging from 1.5× on CPUs to 10×+ on architectures with large SP/DP performance gaps. Run-time memory footprint reductions can halve to quarter overall usage while preserving full double-precision accuracy within solver-imposed condition number bounds (0808.2794, Dunton et al., 2020, Kressner et al., 2023).
• Large-Scale Machine Learning: Deep neural network training uses FP16 for activations, gradients, and forward/backward passes, while retaining FP32 master weights and optimizer states for stability and convergence. Under these paradigms, throughput doubles and memory savings reach 40–50% with negligible accuracy loss for standard benchmarks (Gallouédec, 2021, Lewandowski et al., 2023). In quantized LLM inference, architecture-aware mixed-precision strategies (e.g., spike-aware down-project assignment in LLaMA) recover up to 30 points of perplexity and >10 percentage points of zero-shot accuracy over naive INT8 quantization (Maisonnave et al., 30 Apr 2025).
• Scientific Machine Learning and PDE Solvers: Training PINNs and DeepONets in mixed-precision, with float16 compute and float32 accumulations, matches full-precision accuracy while halving memory and accelerating training by 1.1×–1.9×, provided crucial stability modifications (e.g., Adam $\epsilon$ increase, algebraic loss rewrites) are implemented (Hayford et al., 2024).
• Atmospheric and Climate Modelling: In the GRIST dynamical core, the limited-degree iterative development process benchmarks precision sensitivity per term, enabling conversion of advective fluxes and tracers to single precision, but retaining pressure-gradient and gravity terms at double precision. The result is problem-size independent runtime reduction (24–44%) across a hierarchy from idealized baroclinic waves to climate-scale AMIP runs (Chen et al., 2024).
• Iterative Solvers and Memory-Bound Kernels: New group-shared exponent and segmented mantissa storage methods enable on-the-fly precision adjustment in sparse SpMV, integrating seamlessly into precision-stepped Krylov solvers that switch up precision as convergence stalls. This format achieves 1.2–1.3× end-to-end solver acceleration and robust convergence beyond naive FP16/BF16 schemes (Gao et al., 2024).

5. Adaptive, Multistage, and Precision-Aware Control Policies

Modern mixed-precision paradigms increasingly deploy adaptive and multistage schemes:

• Multistage Iterative Refinement: When early-stage refinement in low precision stalls (due to ill-conditioning or slow convergence), the algorithm escalates to stronger solvers (e.g., GMRES-based updates in higher precision) before refactorizing at higher precision (Oktay et al., 2021). This avoids premature recourse to expensive factorizations, bounding overhead and improving robustness. Switchover criteria rely on local progress measures such as normwise and componentwise corrections, enabling efficient auto-tuning across a continuum of problem conditioning.
• Dynamic Per-Stage Selection: Block-structured pipelines (e.g., FFTMatvec) assign precisions to each stage via Pareto optimization, trading error for runtime and guaranteeing user-specified global accuracy (Venkat et al., 13 Aug 2025). Similar policies appear in tile-level GEMM assignment (Zhang et al., 20 Aug 2025), precision-stepping in iterative Krylov solvers (Guo et al., 7 May 2025, Gao et al., 2024), and attention module maximization in LLM pipelines (Zhang et al., 21 Aug 2025).
• Data-Driven and Model-Specific Allocation: For neural operators and LLMs, precision is assigned based on measured properties such as activation spike magnitude, layer sensitivity, norm ratios, or data normalization. This architecture-aware policy outperforms generalized approaches in both accuracy and efficiency (Maisonnave et al., 30 Apr 2025, Carson et al., 2024).

6. Limitations, Implementation Caveats, and Emerging Research Directions

While mixed-precision paradigms provide substantial acceleration and resource reduction, they require:

• Explicit conditioning constraints. For linear solvers, the matrix condition number (product with unit roundoff of low precision) must remain below unity for convergence. For model reduction, the singular value decay must be fast enough so that low-precision error does not dominate (0808.2794, Dunton et al., 2020).
• Explicit handling of numerically sensitive reductions, boundary conditions, and loss accumulations in high precision, due to catastrophic loss in tiny normal forms under sub-32-bit format (Hayford et al., 2024, Chen et al., 2024).
• Tuning of hyperparameters, such as loss tolerances, thresholds for precision switching, and layer-wise format assignment, sometimes requiring extensive retraining or validation (Maisonnave et al., 30 Apr 2025).
• Specialized memory layouts and packing, particularly for hardware-aware workflows, as correct alignment, tile fragmentation, or coalesced access may be essential to achieve predicted speedups (Zhang et al., 21 Aug 2025, Martínez et al., 13 Jun 2025).

Current and emerging research emphasizes fully adaptive hardware-in-the-loop policies, compiler-driven auto-tuning of per-tensor or per-task precision, and novel floating-point formats (block-shared exponent, custom FP8/4) for both inference and training, as well as the integration of analog computing units (PCM, RRAM) with robust digital correction schemes.

---

Key References:

• "Accelerating Scientific Computations with Mixed Precision Algorithms" (0808.2794)
• "Mixed precision matrix interpolative decompositions for model reduction" (Dunton et al., 2020)
• "Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators" (Abdel-Aziz et al., 2021)
• "Computing $k$-means in mixed precision" (Carson et al., 2024)
• "Mixed-Precision Performance Portability of FFT-Based GPU-Accelerated Algorithms for Block-Triangular Toeplitz Matrices" (Venkat et al., 13 Aug 2025)
• "A mixed precision LOBPCG algorithm" (Kressner et al., 2023)
• "Mixed-Precision In-Memory Computing" (Gallo et al., 2017)
• "Mixed precision in Graphics Processing Unit" (Gallouédec, 2021)
• "Precision-Aware Iterative Algorithms Based on Group-Shared Exponents of Floating-Point Numbers" (Gao et al., 2024)
• "Memory Efficient Mixed-Precision Optimizers" (Lewandowski et al., 2023)
• "Neural Quantum States in Mixed Precision" (Solinas et al., 28 Jan 2026)
• "An Adaptive Mixed Precision and Dynamically Scaled Preconditioned Conjugate Gradient Algorithm" (Guo et al., 7 May 2025)
• "Leveraging Hardware-Aware Computation in Mixed-Precision Matrix Multiply: A Tile-Centric Approach" (Zhang et al., 20 Aug 2025)
• "Precision Where It Matters: A Novel Spike Aware Mixed-Precision Quantization Strategy for LLaMA-based LLMs" (Maisonnave et al., 30 Apr 2025)
• "Mixed-Precision Computing in the GRIST Dynamical Core for Weather and Climate Modelling" (Chen et al., 2024)
• "Multistage Mixed Precision Iterative Refinement" (Oktay et al., 2021)
• "Efficient Mixed-Precision LLM Inference with TurboMind" (Zhang et al., 21 Aug 2025)
• "Speeding up and reducing memory usage for scientific machine learning via mixed precision" (Hayford et al., 2024)
• "The Cambrian Explosion of Mixed-Precision Matrix Multiplication for Quantized Deep Learning Inference" (Martínez et al., 13 Jun 2025)