An Analysis of Variance of the Pantheon+ Dataset: Systematics in the Covariance Matrix?

Abstract: We investigate the statistics of the available Pantheon+ dataset. Noticing that the $\chi^2$ value for the best-fit $\Lambda$CDM model to the real data is small, we quantify how significant its smallness is by calculating the distribution of $\chi^2$ values for the best-fit $\Lambda$CDM model fit to mock Pantheon+-like datasets, using the provided covariance matrix. We further investigate the distribution of the residuals of the Pantheon+ dataset with respect to the best-fit $\Lambda$CDM model, and notice that they scatter less than would be expected from the covariance matrix but find no significant kurtosis. These results point to the conclusion that the Pantheon+ covariance matrix is over-estimated. One simple interpretation of these results is a $\sim$7\% overestimation of errors on SN distance moduli in Pantheon+ data. When the covariance matrix is reduced by subtracting an intrinsic scatter term from the diagonal terms of the covariance matrix, the best-fit $\chi^2$ for the $\Lambda$CDM model achieves a normal value of 1580 and no deviation from $\Lambda$CDM is detected. We further quantify how consistent the $\Lambda$CDM model is with respect to the modified data with the subtracted covariance matrix using model-independent reconstruction techniques such as the iterative smoothing method. We find that the standard model is consistent with the data. There are a number of potential explanations for this smallness of the $\chi^2$, such as a Malmquist bias at high redshift, or accounting for systematic uncertainties by adding them to the covariance matrix, thus approximating systematic uncertainties as statistical ones.