Unified Maximum Likelihood Estimation

• Unified Maximum Likelihood Estimation is a suite of frameworks that generalize classical MLE to address latent variables, unnormalized models, and intractable likelihoods.
• It integrates approaches like h-likelihood, Profile MLE, and maximum approximated likelihood to achieve computational tractability and statistical efficiency.
• The unified methods enhance performance in diverse settings such as regression under uncertainty, large-alphabet inference, and robust divergence estimation.

Unified Maximum Likelihood Estimation (UMLE) encompasses a suite of frameworks that extend or generalize classical maximum likelihood estimation (MLE) to address challenges arising in modern statistical modeling, including unobserved latent variables, intractable likelihoods, large-alphabet settings, unnormalized models, measurement uncertainty, and diverse statistical functionals. These unified treatments aim to deliver statistical efficiency, computational tractability, and finite-sample optimality—often within a single joint (or profile) maximum-likelihood principle. This article synthesizes central methodologies and results from contemporary arXiv literature, outlining key theoretical constructs, computational advances, and unification paradigms in UMLE.

1. Unified Likelihood with Latent and Random Effects: The h-Likelihood

The h-likelihood framework unifies estimation in models with fixed effects, random effects, and missing data by jointly maximizing an extended log-likelihood over both fixed parameters ($\theta$) and latent variables ($u$) (Han et al., 2022). The h-likelihood is defined, with the inclusion of a crucial Jacobian term, as

$$h(\theta, u) = \log f_\theta(y | u) + \log f_\theta(u) + J(u),$$

where $f_\theta(y|u)$ is the observed-data likelihood conditional on $u$, $f_\theta(u)$ is the density of the latent variables, and $J(u) = \log |\partial u/\partial v|$ (with $v$ a canonical reparameterization) ensures correct marginalization at the mode.

Simultaneous estimation is performed by solving the score equations

$$\frac{\partial h}{\partial \theta} = 0, \quad \frac{\partial h}{\partial u} = 0,$$

resulting in ML estimators for both $\theta$ and $u$. The framework generalizes ML imputation: missing data entries are treated as latent $u$, and joint maximization yields “one-shot” mode-based imputations and consistent variance estimates, without recourse to expectation-maximization (EM) algorithms or multiple imputations.

Notably, in linear mixed models, the Jacobian adjustment renders direct maximization valid for variance components estimation—addressing limitations in Henderson’s joint likelihood approach. The h-likelihood thus provides a genuinely unified optimization objective for fixed effects, random effects, variance components, and missing data under a single marginal-likelihood-approximating criterion.

2. Unified Profile and Symmetric Functionals: Profile Maximum Likelihood

Profile Maximum Likelihood (PML) offers a unified, minimax-optimal estimator for a broad class of symmetric distribution functionals and learning tasks under large alphabet or combinatorial regimes (Acharya et al., 2016, Hao et al., 2019). Formally, given $n$ i.i.d. observations from an unknown distribution $p$ over a set $\mathcal X$, define the empirical profile $\phi$ (the histogram of symbol counts, or “fingerprint”). The PML estimator $\hat p_{\text{PML}}$ is

$$\hat p_{\text{PML}} = \arg\max_{q \in \Delta} q(\phi),$$

where $q(\phi)$ is the probability of observing the profile $\phi$ under $q$.

This plug-in principle yields sample-optimal estimators for:

• Distribution estimation under sorted $\ell_1$ distance: $n \asymp \frac{k}{\epsilon^2 \log k}$ samples for alphabet size $k$ and accuracy $\epsilon$.
• Additive symmetric functions: PML plug-in achieves error within a constant factor of the best estimator for properties like entropy, support size, or coverage, with exponential concentration.
• Rényi and Shannon entropy, support coverage, and earth-mover metrics, as well as identity and uniformity testing.
• The truncated PML (TPML) and approximate PML (APML) variants enable practical, near-linear-time computation with minimal degradation in statistical benchmarks (Hao et al., 2019).

Theoretical justification leverages profile sufficiency and competitive optimality: for any symmetric property, the PML plug-in’s performance is within a small multiplicative overhead of the best possible profile-based estimator, often matching formal minimax rates.

3. Unified Frameworks for Intractable and Unnormalized Likelihoods

Models with intractable or unnormalized likelihoods—such as Markov random fields or energy-based models—are unified under approximated or self-normalized likelihood frameworks (Griebel et al., 2019, Uehara et al., 2019). The Maximum Approximated Likelihood (MAL) approach considers a sequence of increasingly accurate likelihood approximations $\tilde f_r(x;\theta)$ (e.g., Monte Carlo, quasi–Monte Carlo, Gaussian quadrature, sparse grids), and sets

$$\hat \theta_{\mathrm{MAL}} = \arg\max_\theta \sum_{i=1}^n \log \tilde f_{R(n)}(x_i; \theta),$$

with $R(n)$ selected to ensure $\sqrt n\, \mathcal E(R(n)) \to 0$, where $\mathcal E(r)$ controls uniform approximation error in function and derivatives. This encompasses maximum simulated likelihood, quadrature methods, and sparse-grid integrators. Under mild regularity, MAL estimators possess consistency and asymptotic normality matching standard MLE (Griebel et al., 2019).

For unnormalized models, the self-density-ratio matching estimator (SDRME) unifies Bregman-divergence-based density ratio matching with nonparametric estimation. The objective

$$L_n(\theta) = -\frac{1}{n} \sum_{i=1}^n \log q(x_i;\theta) + \log \left\{ \frac{1}{n} \sum_{i=1}^n \frac{q(x_i;\theta)}{\hat p_n(x_i)} \right\}$$

achieves Fisher-information-efficient estimation without recourse to evaluation or differentiation of the normalizing constant $Z(\theta)$, yielding $\mathcal O(n)$ per-iteration cost and statistically optimal efficiency (Uehara et al., 2019).

4. Unified Likelihood in Regression Under Measurement Uncertainty

The generalized RV-ML (GRV-ML) estimator delivers a unified, convex-optimization-based maximum-likelihood solution for linear regression under Gaussian measurement matrix uncertainty, accommodating both underdetermined and overdetermined systems and rank-deficient design matrices (Guo et al., 15 Jul 2025). The joint negative log-likelihood

$$f(y;x) = \frac{\|y - Hx\|^2}{2 (\sigma_e^2 \|x\|^2 + \sigma_\epsilon^2)} + \frac{M}{2} \log(\sigma_e^2 \|x\|^2 + \sigma_\epsilon^2)$$

is lifted to a convex program in auxiliary variables, yielding unique solutions via SVD and one-dimensional bisection. Noteworthy is the result that, in underdetermined regimes, additional measurement noise randomness can be beneficial, enabling finite-variance point estimation where ordinary least squares fails.

5. Parameter-Free and Distributional Unification: KL Projection and Distributional Expectation

A parameter-free, distribution-valued unification is offered by considering the random estimator as a random distribution and using KL-divergence as loss (Vos et al., 2015). Within this framework:

• The (distributional) expectation $E^\dagger[\mathbf R]$ and variance $V^\dagger(\mathbf R)$ mirror ordinary MSE decomposition for KL risk.
• Rao–Blackwellization extends to distribution-valued estimators.
• In exponential families, the classical MLE is unique in uniformly minimizing the KL variance among all unbiased estimators, and retains this property even when the model is misspecified, provided the KL-projection of the true law onto the parametric manifold is used.
• This framework covers both finite-sample and infinite-dimensional cases, underpinning the parameter-invariance and robustness of unified likelihood estimation.

6. Unified Perspective on Information Divergences via Likelihood

Maximum likelihood density ratio estimation (DRE) unifies the KL-divergence and integral probability metric (IPM) families under a single likelihood-based functional (Kato et al., 2022). For (possibly weighted) log-likelihoods of parametric density ratios between samples from $P$ and $Q$, one obtains:

• For standard sampling, the population objective maximizes $D_{\mathrm{KL}}(P\|Q)$ or $D_{\mathrm{KL}}(Q\|P)$.
• For stratified (balanced) sampling, the objective recovers IPMs.
• Varying a weight parameter $\lambda \in [0,1]$ yields the Density Ratio Metric (DRM) family $D_{\mathrm{DRM}}^\lambda(P\|Q)$, bridging KL and IPM divergences in a unified family, with smooth interpolation between both extremes.

This unified approach supports both theoretical analysis and practical algorithms for density-ratio estimation and generative modeling (e.g., SLoGAN GAN training with improved stability).

7. Unified Characterization Theorems for Maximum Likelihood Estimation

MLE characterization theorems, originally developed for specific distributional families, are unified under general results for one-parameter transformation groups (Duerinckx et al., 2012). The framework introduces:

• A general expression of the likelihood for group-family models: $f_H(x;\theta) = H'_\theta(x) f(H_\theta(x))$.
• The notion of MLE equivalence classes: two densities sharing proportional group-scores are MLE-indistinguishable.
• Minimal necessary sample size (MNSS) required for characterization, formalized via the minimal covering sample size (MCSS).
• Master theorems quantifying necessary and sufficient conditions for parametrically unique MLE identification, extending classic Gaussian, exponential, and gamma characterizations to broad continuous families and new cases (e.g., skewness parameters).

These results provide a decision-theoretic recipe for verifying when a unified likelihood principle uniquely determines a statistical model within a wider class.

---

Unified maximum likelihood estimation thus represents a convergence of principal statistical techniques—marginalization and profiling for latent variables, plug-in methods for properties over distributions, likelihood-based estimation in intractable models, and parameter-free or information-geometric characterizations—into structurally and computationally unified frameworks. These advances result in improved adaptivity, computational tractability, and optimality guarantees for a wide spectrum of contemporary inference problems.