Fair Box ordinate transform for forecasts following a multivariate Gaussian law

Abstract: Monte Carlo techniques are the method of choice for making probabilistic predictions of an outcome in several disciplines. Usually, the aim is to generate calibrated predictions which are statistically indistinguishable from the outcome. Developers and users of such Monte Carlo predictions are interested in evaluating the degree of calibration of the forecasts. Here, we consider predictions of $p$-dimensional outcomes sampling a multivariate Gaussian distribution and apply the Box ordinate transform (BOT) to assess calibration. However, this approach is known to fail to reliably indicate calibration when the sample size n is moderate. For some applications, the cost of obtaining Monte-Carlo estimates is significant, which can limit the sample size, for instance, in model development when the model is improved iteratively. Thus, it would be beneficial to be able to reliably assess calibration even if the sample size n is moderate. To address this need, we introduce a fair, sample size- and dimension-dependent version of the Gaussian sample BOT. In a simulation study, the fair Gaussian sample BOT is compared with alternative BOT versions for different miscalibrations and for different sample sizes. Results confirm that the fair Gaussian sample BOT is capable of correctly identifying miscalibration when the sample size is moderate in contrast to the alternative BOT versions. Subsequently, the fair Gaussian sample BOT is applied to two to 12-dimensional predictions of temperature and vector wind using operational ensemble forecasts of the European Centre for Medium-Range Weather Forecasts (ECMWF). Firstly, perfectly reliable situations are considered where the outcome is replaced by a forecast that samples the same distribution as the members in the ensemble. Secondly, the BOT is computed using estimates of the actual temperature and vector wind from ECMWF analyses.