On Semi-supervised Estimation of Discrete Distributions under f-divergences
Abstract: We study the problem of estimating the joint probability mass function (pmf) over two random variables. In particular, the estimation is based on the observation of $m$ samples containing both variables and $n$ samples missing one fixed variable. We adopt the minimax framework with $lp_p$ loss functions. Recent work established that univariate minimax estimator combinations achieve minimax risk with the optimal first-order constant for $p \ge 2$ in the regime $m = o(n)$, questions remained for $p \le 2$ and various $f$-divergences. In our study, we affirm that these composite estimators are indeed minimax optimal for $lp_p$ loss functions, specifically for the range $1 \le p \le 2$, including the critical $l_1$ loss. Additionally, we ascertain their optimality for a suite of $f$-divergences, such as KL, $\chi2$, Squared Hellinger, and Le Cam divergences.
- A. Wald, “Statistical decision functions,” The Annals of Mathematical Statistics, vol. 20, no. 2, pp. 165–205, 1949. [Online]. Available: http://www.jstor.org/stable/2236853
- S. Trybula, “Some problems of simultaneous minimax estimation,” The Annals of Mathematical Statistics, vol. 29, no. 1, pp. 245–253, 1958.
- I. Olkin and M. Sobel, “Admissible and minimax estimation for the multinomial distribution and for k independent binomial distributions,” The Annals of Statistics, vol. 7, no. 2, Mar. 1979. [Online]. Available: https://doi.org/10.1214/aos/1176344613
- M. Wilczyński, “Minimax estimation for the multinomial and multivariate hypergeometric distributions,” Sankhyā: The Indian Journal of Statistics, Series A, pp. 128–132, 1985.
- D. Braess and T. Sauer, “Bernstein polynomials and learning theory,” Journal of Approximation Theory, vol. 128, no. 2, pp. 187–206, 2004.
- Y. Han, J. Jiao, and T. Weissman, “Minimax estimation of discrete distributions under l1subscript𝑙1l_{1}italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT loss,” CoRR, vol. abs/1411.1467, 2014. [Online]. Available: http://arxiv.org/abs/1411.1467
- S. Kamath, A. Orlitsky, D. Pichapati, and A. T. Suresh, “On learning distributions from their samples,” in Proceedings of The 28th Conference on Learning Theory, ser. Proceedings of Machine Learning Research, P. Grünwald, E. Hazan, and S. Kale, Eds., vol. 40. Paris, France: PMLR, 03–06 Jul 2015, pp. 1066–1100. [Online]. Available: https://proceedings.mlr.press/v40/Kamath15.html
- H. S. Melihcan Erol, E. Sula, and L. Zheng, “On semi-supervised estimation of distributions,” in 2023 IEEE International Symposium on Information Theory (ISIT). IEEE, Jun. 2023.
- F. Chung and L. Lu, “Connected components in random graphs with given expected degree sequences,” Annals of Combinatorics, vol. 6, no. 2, pp. 125–145, Nov. 2002. [Online]. Available: https://doi.org/10.1007/pl00012580
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.