On the universal calibration of Pareto-type linear combination tests

Abstract: It is often of interest to test a global null hypothesis using multiple, possibly dependent, $p$-values by combining their strengths while controlling the Type I error. Recently, several heavy-tailed combinations tests, such as the harmonic mean test and the Cauchy combination test, have been proposed: they map $p$-values into heavy-tailed random variables before combining them in some fashion into a single test statistic. The resulting tests, which are calibrated under the assumption of independence of the $p$-values, have shown to be rather robust to dependence. The complete understanding of the calibration properties of the resulting combination tests of dependent and possibly tail-dependent $p$-values has remained an important open problem in the area. In this work, we show that the powerful framework of multivariate regular variation (MRV) offers a nearly complete solution to this problem. We first show that the precise asymptotic calibration properties of a large class of homogeneous combination tests can be expressed in terms of the angular measure -- a characteristic of the asymptotic tail-dependence under MRV. Consequently, we show that under MRV, the Pareto-type linear combination tests, which are equivalent to the harmonic mean test, are universally calibrated regardless of the tail-dependence structure of the underlying $p$-values. In contrast, the popular Cauchy combination test is shown to be universally honest but often conservative; the Tippet combination test, while being honest, is calibrated if and only if the underlying $p$-values are tail-independent. One of our major findings is that the Pareto-type linear combination tests are the only universally calibrated ones among the large family of possibly non-linear homogeneous heavy-tailed combination tests.