Noise-marginalized optimal statistic: A robust hybrid frequentist-Bayesian statistic for the stochastic gravitational-wave background in pulsar timing arrays

Abstract: Observations have revealed that nearly all galaxies contain supermassive black holes (SMBHs) at their centers. When galaxies merge, these SMBHs form SMBH binaries (SMBHBs) that emit low-frequency gravitational waves (GWs). The incoherent superposition of these sources produce a stochastic GW background (GWB) that can be observed by pulsar timing arrays (PTAs). The optimal statistic is a frequentist estimator of the amplitude of the GWB that specifically looks for the spatial correlations between pulsars induced by the GWB. In this paper, we introduce an improved method for computing the optimal statistic that marginalizes over the red noise in individual pulsars. We use simulations to demonstrate that this method more accurately determines the strength of the GWB, and we use the noise-marginalized optimal statistic to compare the significance of monopole, dipole, and Hellings-Downs (HD) spatial correlations and perform sky scrambles.