Improved Sublinear-time Moment Estimation using Weighted Sampling

Abstract: In this work we study the {\it moment estimation} problem using weighted sampling. Given sample access to a set $A$ with $n$ weighted elements, and a parameter $t>0$, we estimate the $t$-th moment of $A$ given as $S_t=\sum_{a\in A} w(a)^t$. For t=1, this is the sum estimation problem. The moment estimation problem along with a number of its variants have been extensively studied in streaming, sublinear and distributed communication models. Despite being well studied, we don't yet have a complete understanding of the sample complexity of the moment estimation problem in the sublinear model and in this work, we make progress on this front. On the algorithmic side, our upper bounds match the known upper bounds for the problem for $t>1$. To the best of our knowledge, no sublinear algorithms were known for this problem for $0<t\<1$. We design a sublinear algorithm for this problem for $t\>1/2$ and show that no sublinear algorithms exist for $t\leq 1/2$. We prove a $\Omega(\frac{n^{1-1/t}\ln 1/\delta}{\epsilon^2})$ lower bound for moment estimation for $t>1$, and show optimal sample complexity bound $\Theta(\frac{n^{1-1/t}\ln 1/\delta}{\epsilon^2})$ for moment estimation for $t\geq 2$. Hence, we obtain a complete understanding of the sample complexity for moment estimation using proportional sampling for $t\geq 2$. We also study the moment estimation problem in the beyond worst-case analysis paradigm and identify a new {\it moment-density} parameter of the input that characterizes the sample complexity of the problem using proportional sampling and derive tight sample complexity bounds with respect to that parameter. We also study the moment estimation problem in the hybrid sampling framework in which one is given additional access to a uniform sampling oracle and show that hybrid sampling framework does not provide any additional gain over the proportional sampling oracle in the worst case.