Interpreting Internal Consistency of DES Measurements

Abstract: Bayesian evidence ratios are widely used to quantify the statistical consistency between different experiments. However, since the evidence ratio is prior dependent, the precise translation between its value and the degree of concordance/discordance requires additional information. The most commonly adopted metric, the Jeffreys scale, can falsely suggest agreement between datasets when priors are chosen to be sufficiently wide. In this work, we examine evidence ratios in a DES-Y1 simulated analysis, focusing on the internal consistency between weak lensing and galaxy clustering. We study two scenarios using simulated data in controlled experiments. First, we calibrate the expected evidence ratio distribution given noise realizations around the best fit DES-Y1 $\Lambda$CDM cosmology. Second, we show the behavior of evidence ratios for noiseless fiducial data vectors simulated using a modified gravity model, which generates internal tension in the $\Lambda$CDM analysis. We show that the choice of prior could conceal the discrepancies between weak lensing and galaxy clustering induced by such models and that the evidence ratio in a DES-Y1 study is, indeed, biased towards agreement.