Exponent-Mantissa Bit Ratio in FP Formats

• Exponent-mantissa bit ratio is a key parameter in floating-point representations that allocates bits between exponent and mantissa, balancing dynamic range and precision.
• Empirical studies reveal that optimal splits, such as FP8 formats with ratios like 4:3 or 5:3, consistently reduce validation loss in neural network training.
• Adaptive and tapered formats dynamically adjust the exponent and mantissa bits based on data distribution and task requirements to enhance efficiency and accuracy.

The exponent-mantissa bit ratio is a fundamental parameter in floating-point (FP) number representation, determining how the available bit budget for a floating-point value is divided between encoding dynamic range (the exponent) and local precision (the mantissa or significand). The optimal split profoundly affects both the numerical behavior and practical performance of low-precision computing, impacting neural network accuracy, resource utilization, robustness, and hardware efficiency across diverse applications.

1. Mathematical Foundations of Exponent-Mantissa Splitting

A normalized n-bit floating-point value $f$ is typically represented as:

$$f = (\pm 1) \times 2^{E - \text{bias}} \times (1 + m)$$

• $s$ : sign bit
• $E$ : exponent field (width $e$ bits, unsigned integer)
• $m$ : mantissa (width $m$ bits, fractional part in binary)
• bias : an integer offset (usually $2^{e-1}-1$ for IEEE-754 formats)

Given $n = 1 + e + m$ (1 for sign), the exponent-mantissa bit ratio $r = e/m$ directly balances dynamic range against precision.

Key principles:

• Increasing $$f = (\pm 1) \times 2^{E - \text{bias}} \times (1 + m)$$0 increases representable orders of magnitude but coarsens quantization steps.
• Increasing $$f = (\pm 1) \times 2^{E - \text{bias}} \times (1 + m)$$1 sharpens local quantization (reduces unit-in-last-place (ulp)), but narrows the overall dynamic range.

A single-bit shift between exponent and mantissa multiplies or divides the dynamic range and relative precision by 2, highlighting the exponential sensitivity of this allocation (Kuzmin et al., 2022).

2. Empirical Optimization and Scaling Laws

Extensive empirical studies have established scaling laws for the optimal allocation of exponent and mantissa bits. The unified scaling law for FP quantization performance in LLM training expresses validation loss as

$$f = (\pm 1) \times 2^{E - \text{bias}} \times (1 + m)$$2

with

$$f = (\pm 1) \times 2^{E - \text{bias}} \times (1 + m)$$3

Fitted exponents for 366 full pre-training runs give $$f = (\pm 1) \times 2^{E - \text{bias}} \times (1 + m)$$4 for exponent bits and $$f = (\pm 1) \times 2^{E - \text{bias}} \times (1 + m)$$5 for mantissa bits. Since $$f = (\pm 1) \times 2^{E - \text{bias}} \times (1 + m)$$6, increasing exponent bits slightly more than mantissa bits consistently reduces loss (Sun et al., 5 Jan 2025).

Given bit budget $$f = (\pm 1) \times 2^{E - \text{bias}} \times (1 + m)$$7, the analytically optimal split is:

$$f = (\pm 1) \times 2^{E - \text{bias}} \times (1 + m)$$8

This results in exponent:mantissa splits of roughly

• FP4: $$f = (\pm 1) \times 2^{E - \text{bias}} \times (1 + m)$$9
• FP8: $s$0 or $s$1
• BF16: $s$2
• General rule: assign $s$352% of non-sign bits to exponent (Sun et al., 5 Jan 2025).

3. Distributional Sensitivity and Task Dependence

Optimal exponent-mantissa bit ratio is sensitive to the data’s distributional properties:

• Light-tailed (Gaussian): More mantissa bits minimize mean squared error (MSE); $s$4M$s$5E or $s$6M$s$7E (e.g., weights, activations in CNNs) (Kuzmin et al., 2022).
• Heavy-tailed (Student’s t, transformers): More exponent bits are required to absorb outliers; $s$8M$s$9E or $E$0M$E$1E (e.g., transformer activations) (Kuzmin et al., 2022).
• Regression, non-classification tasks: Certain tasks, such as speech enhancement, permit mantissa to be driven nearly to zero with negligible loss (Hsu et al., 2018).

For elementwise quantized convolutions in the MLS format, CIFAR-10 is robust to as low as $E$2 exponent bits, $E$3 mantissa bit without $E$4 accuracy loss, while ImageNet requires $E$5 exponent bits and $E$6 mantissa bits (Zhong et al., 2020).

4. Architectures, Formats, and Adaptive Strategies

Fixed-format Examples

Table: Representative floating-point formats and exponent-mantissa splits.

Format (total $E$7)	Exponent bits	Mantissa bits	Ratio $E$8	Use case
E2M1 (FP4)	2	1	2.0	LLMs, very low-prec.
E4M3 (FP8)	4	3	1.33	Activations, weights (Micikevicius et al., 2022)
E5M2 (FP8)	5	2	2.5	Gradients, tails (Micikevicius et al., 2022)
BF16	8	7	1.14	General training (Popescu et al., 2021)
1/6/9 (16-bit)	6	9	0.67	Mixed-precision NN (Popescu et al., 2021)

Adaptive, Tapered, and Flexible Formats

Modern approaches include:

• Tapered precision (HiFloat8): Vary mantissa down as exponent magnitude grows; in HiF8, central exponents use $E$9 bits mantissa, outer tails $e$0–$e$1 bits, maximizing precision where typical values lie (Luo et al., 2024).
• Floating-Floating-Point (F2P): Hyper-exponent field per-value dynamically determines exponent-mantissa split, giving sub-range-variable precision or dynamic-range prioritization (SR/LI modes) (Cohen et al., 2024).
• Adaptive learning (Quantum Mantissa/Exponent/BitWave): Layerwise or tensorwise $e$2 are learned via backprop or statistical trends, typically yielding %%%%43$e$44%%%%1 (activations) or $e$5:$e$6 (weights) allocation in ResNet-18/ImageNet (Nikolić et al., 2022).

5. Impact on Quantization Error, Robustness, and Hardware

The error structure in floating-point quantization is determined by $e$7:

• Grid step spacing in $e$8: $e$9
• Relative quantization error (floating): $m$0, uniform across the dynamic range (Kuzmin et al., 2022).

Larger $m$1 cushions overflow/underflow in distributed representations or under outlier exposure, while $m$2 ensures that signal-to-quantization-noise ratio (SQNR) remains high in “center” values. For quantum control, the exponent-mantissa split must also account for bit-flip sensitivity in control electronics. For instance, error expectations require $m$3 to constrain worst-case total variation deviation $m$4 under single-bit flips (Das et al., 2024).

Energy efficiency follows: minimizing $m$5 permits smaller (and therefore more energy-efficient) adders and multipliers (Zhong et al., 2020), while increasing $m$6 (with managed clipping) ensures no catastrophic overflow.

6. Compression, Post-training Quantization, and Error Correction

In aggressive model compression, the exponent-only floating-point quantized neural network (EOFP-QNN) can, for speech enhancement, drive the mantissa to $m$7 (all resolution in exponent), with exponent field re-biased to the narrowest observed range, achieving model size reductions to $m$8 with $m$9 performance drop (Hsu et al., 2018).

Dynamic tuning strategies—such as layerwise learning of bit-allocations—outperform static, globally assigned formats, and can reach $m$0–$m$1 compression with negligible accuracy loss, as in Quantum Mantissa/Quantum Exponent (Nikolić et al., 2022).

7. Practical Guidelines and Format Selection

Principled design rules emerging from the literature include:

• Sub-8-bit FP: Allocate slightly more bits to exponent ($m$2 of non-sign bits) than mantissa, e.g., FP8 as $m$3:$m$4 or $m$5:$m$6 (Sun et al., 5 Jan 2025).
• Precision scheduling: Low mantissa (1–2 bits) is tolerable in small-scale or light-tailed tasks, but large-scale tasks and outlier-prone distributions require more $m$7.
• Denormals: Sufficient exponent bits reduce the need for subnormal support; this allows hardware to safely flush denormals to zero and maximize throughput (Popescu et al., 2021).
• Tapered formats and hyper-exponent/“dot” fields (as in HiFloat8, F2P) provide a continuum of allocation, and outperform rigid splits especially for federated learning and network measurement (Luo et al., 2024, Cohen et al., 2024).

• "FP8 Quantization: The Power of the Exponent" (Kuzmin et al., 2022)
• "Scaling Laws for Floating Point Quantization Training" (Sun et al., 5 Jan 2025)
• "Representation range needs for 16-bit neural network training" (Popescu et al., 2021)
• "FP8 Formats for Deep Learning" (Micikevicius et al., 2022)
• "Ascend HiFloat8 Format for Deep Learning" (Luo et al., 2024)
• "Floating-floating point: a highly accurate number representation with flexible Counting ranges" (Cohen et al., 2024)
• "Exploring the Potential of Low-bit Training of Convolutional Neural Networks" (Zhong et al., 2020)
• "Investigating impact of bit-flip errors in control electronics on quantum computation" (Das et al., 2024)
• "Schrödinger's FP: Dynamic Adaptation of Floating-Point Containers for Deep Learning Training" (Nikolić et al., 2022)
• "A study on speech enhancement using exponent-only floating point quantized neural network (EOFP-QNN)" (Hsu et al., 2018)