Bootstrap inference after using multiple queries for model selection

Abstract: In this work, we provide a refinement of the selective CLT result of Tian and Taylor (2015), which allows for selective inference in non-parametric settings by adjusting for the asymptotic Gaussian limit for selection. Under some regularity assumptions on the density of the randomization, including heavier tails than Gaussian satisfied by e.g. logistic distribution, we prove the selective CLT holds without any assumptions on the underlying parameter, allowing for rare selection events. We also show a selective CLT result for Gaussian randomization, though the quantitative results are qualitatively different for the Gaussian randomization as compared to the heavier tailed results. Furthermore, we propose a bootstrap version of this test statistic, which is provably asymptotically pivotal uniformly across a family of non-parametric distributions. This result can be interpreted as resolving the impossibility results of Leeb and Potscher (2006). We describe several sampling methods involving the projected Langevin Monte Carlo to compute the bootstrapped test statistic and the corresponding confidence intervals valid after selection. The applications of our work include valid inferential and sampling tools after running various model selection algorithms including their combinations into multiple views/queries framework. We also present a way to do data carving, providing more powerful tests than classical data splitting by reusing the information in the data from the first stage.