Markov Chain Monte Carlo Significance Tests

Abstract: Monte Carlo significance tests are a general tool that produce p-values by generating samples from the null distribution. However, Monte Carlo tests are limited to null hypothesis which we can exactly sample from. Markov chain Monte Carlo (MCMC) significance tests are a way to produce statistical valid p-values for null hypothesis we can only approximately sample from. These methods were first introduced by Besag and Clifford in 1989 and make no assumptions on the mixing time of the MCMC procedure. Here we review the two methods of Besag and Clifford and introduce a new method that unifies the existing procedures. We use simple examples to highlight the difference between MCMC significance tests and standard Monte Carlo tests based on exact sampling. We also survey a range of contemporary applications in the literature including goodness-of-fit testing for the Rasch model, tests for detecting gerrymandering [8] and a permutation based test of conditional independence [3].