Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation

Abstract: Mutual information (MI) estimation is a central problem in machine learning and statistics; however, existing benchmarks typically evaluate estimators on simplified, low-dimensional distributions, leaving their performance on complex, realistic data largely unexplored. We address this gap with a comprehensive benchmarking framework grounded in a unified copula-theoretic perspective that subsumes existing benchmarks as special cases. Within this framework, we propose two complementary families of tests: a copula-first family that systematically varies ground-truth MI, dimensionality, and marginal complexity using synthetic and flow-based transformations; and a marginals-first family that couples real-world image data with controlled dependency structures, extending the classic same-class-pairing paradigm. We use this suite to extensively evaluate three classes of estimators: non-parametric, discriminative, and generative. Contrary to prevailing assumptions, our results indicate that there is no universal winner: each category can systematically outperform all other estimators under specific setups. By analyzing these cases, we identify fundamental estimation barriers and propose new tests that more effectively stress these specific limitations. We share the open source code at https://github.com/VanessB/mutinfo.

• The paper introduces a copula-based framework that unifies synthetic and real-world MI estimator evaluations.
• It rigorously compares non-parametric, discriminative, and generative estimators, highlighting regime-specific strengths and weaknesses.
• The study reveals fundamental estimation barriers such as exponential sample complexity and numerical instability impacting MI estimation.

Authoritative Summary of "Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation" (2607.03487)

Motivation and Problem Statement

Mutual information (MI) estimation is foundational in machine learning, underpinning representation learning, generative modeling, feature selection, and hypothesis testing. However, MI estimators—ranging from classical non-parametric to neural-based and diffusion-inspired models—have been conventionally evaluated on restricted, often low-dimensional synthetic distributions. These limited benchmarks fail to rigorously probe estimator performance on more complex, high-dimensional, or real-world data. The paper introduces a unified copula-theoretic perspective to subsume existing benchmarking paradigms and, crucially, proposes a comprehensive benchmarking suite that exposes estimator strengths, weaknesses, and estimation barriers across both synthetic and realistic domains.

Benchmarking Framework and Unified Theory

The central theoretical contribution is the adoption of a copula-based view. Copulas disentangle marginal complexities from dependency structures, permitting principled control over MI, dimensionality, and marginal complexity. The authors formalize two complementary benchmark families:

• Copula-first family: Synthetic experiments systematically vary ground-truth MI, dimensionality, and marginal distributions using flow-based transformations, enabling controlled stress-testing of estimators (Figure 1).

Figure 1: Rectangles—synthetic high-dimensional images constructed via copula-driven invertible flow-based transformations.

• Marginals-first family: Real-world data (e.g., MNIST, CIFAR-10) is coupled using controlled dependency structures, extending same-class pairing to general stochastic coupling, thereby evaluating estimator robustness on realistic marginals with analytically tractable ground-truth MI.

This unified framework reveals that MI challenges arise from two sources: complex copulas (dependency structures) and complex marginal distributions, and permits systematic decomposition and analysis across these axes.

Classes of MI Estimators and Their Structural Limitations

The paper extensively evaluates three principal estimator categories:

• Non-parametric: k-NN-based (KSG, WKL), kernel density estimation, histogram-based methods. These are strong in low dimensions but break down in high-dimensional or complex distribution settings.
• Discriminative: Neural classifier-based (MINE, NWJ, InfoNCE), learning density ratios directly. Universally applicable but exhibit saturation and instability at high MI or high dimensionality. Notably, InfoNCE’s quadratic complexity renders it impractical for high-dimensional raw data without embedding compression.
• Generative: Modern normalizing flows, diffusion-based (MINDE, InfoBridge). These permit tractable MI estimation in highly complex settings but demand expensive modeling and introduce smoothing artifacts, especially with perturbed data.

Contrary to prevailing assumptions, the authors document that no estimator is universally superior. Each class excels under distinct regimes defined by dimensionality, MI magnitude, and marginal or copula complexity.

Empirical Evaluation: Synthetic and Real-World Benchmarks

The experimental suite provides rigorous comparative analysis:

Synthetic Continuous and Discrete Benchmarking

• Across continuous (Gaussian, Student’s t, smoothed uniform, LGE, rare event channel) and discrete distributions, generative estimators (particularly diffusion-based MINDE) demonstrate superior accuracy at high MI and dimensionality (Figure 2, Figure 3).

  Figure 2: MI estimation accuracy curves for all estimator families on synthetic continuous distributions, highlighting generative superiority at high MI.

  

  Figure 3: Aggregated comparison of estimator families on continuous synthetic distributions, revealing regions where k-NN, variational, and generative approaches each dominate.

• k-NN-based estimators outperform neural methods in low-dimensional, low-MI regimes and exhibit best results with small $k$, aligning with reduced bias–variance trade-off.

Flow-Based and Copula-Grounded Synthetics

• Flow-based copula transformations drastically increase benchmark difficulty, exposing limitations in discriminative estimators and requiring generative models to handle non-trivial dependency morphisms (Figure 1).

Real Data with Synthetic Copulas

• MNIST and CIFAR-10 experiments using same-class pairing and controlled attribute regression (e.g., brightness scaling) show that neural discriminative estimators are robust to embedding dimension but fail on raw high-dimensional data.
• Generative diffusion-based estimators (MINDE) outperform in high MI settings, but parameterization and numerical instability can cause estimator breakdown, especially in extreme high dimension, as demonstrated by MINDE-J (Table: MNIST/CIFAR-10).

Analysis of Estimation Barriers and Failure Modes

The authors identify and formalize four fundamental MI estimation barriers:

1. Exponential sample complexity: Estimation sample requirement grows exponentially both in MI and dimensionality, and formal bounds cap reliable MI estimation by $\log N$ for $N$ samples.
2. Numerical instability: Even in Gaussian settings, estimator accuracy is tightly constrained by floating-point precision when MI is high, reflected in substantial errors for large $\rho$ (correlation coefficient).
3. High intrinsic variance: PMI’s variance is distribution-dependent, propagating into unstable MI estimates—a property present even in exact estimators due to intrinsic randomness of dependency structures.
4. Sensitivity to noise: Diffusion-based methods inherently smooth sharp distributions during training, leading to rapid MI decay with minimal perturbations, posing a challenge for highly concentrated data.

   Figure 4: MI estimation for discrete experiments (all estimators), demonstrating variance and instability in high-MI regimes.

   

   Figure 5: Aggregated analysis of estimator families on discrete synthetic distributions, further confirming distribution-dependent estimator failure modes.

Practical Implications and Guidance

• Generative MI estimators are necessary for high-dimensional, high-MI, or real data tasks but incur substantial modeling and computational overhead.
• Neural discriminative estimators are recommended for learned (compressed) embeddings, not raw data.
• Non-parametric k-NN methods remain optimal for low-dimension, low-MI, or small-sample scenarios.

Compression and preliminary dimensionality reduction are shown to be universally beneficial, not only for classical estimators but also for neural approaches.

Conclusion

This work delivers a rigorous, copula-grounded framework and benchmark suite for MI estimation, exposing intricately structured practical and theoretical barriers. The comparative evaluation demonstrates that estimator efficacy is highly regime-dependent, negating the viability of universal solutions. The paper provides actionable estimator selection guidance and exposes critical trade-offs that must be addressed to unlock robust MI estimation for complex or realistic machine learning tasks.

Future developments should focus on bridging modeling-complexity trade-offs, improving estimator robustness to intrinsic data variance, and mitigating smoothing artifacts in generative approaches. The presented benchmark suite sets a baseline for principled advancement in MI estimation and information-theoretic analysis.