Euclid preparation. Accurate and precise data-driven angular power spectrum covariances

Abstract: We develop techniques for generating accurate and precise internal covariances for measurements of clustering and weak lensing angular power spectra. These methods are designed to produce non-singular and unbiased covariances for Euclid's large anticipated data vector and will be critical for validation against observational systematic effects. We construct jackknife segments that are equal in area to high precision by adapting the binary space partition algorithm to work on arbitrarily shaped regions on the unit sphere. Jackknife estimates of the covariances are internally derived and require no assumptions about cosmology or galaxy population and bias. Our covariance estimation, called DICES (Debiased Internal Covariance Estimation with Shrinkage), first estimates a noisy covariance through conventional delete-1 jackknife resampling. This is followed by linear shrinkage of the empirical correlation matrix towards the Gaussian prediction, rather than linear shrinkage of the covariance matrix. Shrinkage ensures the covariance is non-singular and therefore invertible, critical for the estimation of likelihoods and validation. We then apply a delete-2 jackknife bias correction to the diagonal components of the jackknife covariance that removes the general tendency for jackknife error estimates to be biased high. We validate internally derived covariances, which use the jackknife resampling technique, on synthetic Euclid-like lognormal catalogues. We demonstrate that DICES produces accurate, non-singular covariance estimates, with the relative error improving by $33\%$ for the covariance and $48\%$ for the correlation structure in comparison to jackknife estimates. These estimates can be used for highly accurate regression and inference.