Differentially Private Kolmogorov-Smirnov-Type Tests

Abstract: Hypothesis testing is a central problem in statistical analysis, and there is currently a lack of differentially private tests which are both statistically valid and powerful. In this paper, we develop several new differentially private (DP) nonparametric hypothesis tests. Our tests are based on Kolmogorov-Smirnov, Kuiper, Cram\'er-von Mises, and Wasserstein test statistics, which can all be expressed as a pseudo-metric on empirical cumulative distribution functions (ecdfs), and can be used to test hypotheses on goodness-of-fit, two samples, and paired data. We show that these test statistics have low sensitivity, requiring minimal noise to satisfy DP. In particular, we show that the sensitivity of these test statistics can be expressed in terms of the base sensitivity, which is the pseudo-metric distance between the ecdfs of adjacent databases and is easily calculated. The sampling distribution of our test statistics are distribution-free under the null hypothesis, enabling easy computation of $p$-values by Monte Carlo methods. We show that in several settings, especially with small privacy budgets or heavy-tailed data, our new DP tests outperform alternative nonparametric DP tests.