Robust Statistics meets elicitability: When fair model validation breaks down

Abstract: A crucial part of data analysis is the validation of the resulting estimators, in particular, if several competing estimators need to be compared. Whether an estimator can be objectively validated is not a trivial property. If there exists a loss function such that the theoretical risk is minimized by the quantity of interest, this quantity is called elicitable, allowing estimators for this quantity to be objectively validated and compared by evaluating such a loss function. Elicitability requires assumptions on the underlying distributions, often in the form of regularity conditions. Robust Statistics is a discipline that provides estimators in the presence of contaminated data. In this paper, we, introducing the elicitability breakdown point, formally pin down why the problems that contaminated data cause for estimation spill over to validation, letting elicitability fail. Furthermore, as the goal is usually to estimate the quantity of interest w.r.t. the non-contaminated distribution, even modified notions of elicitability may be doomed to fail. The performance of a trimming procedure that filters out instances from non-ideal distributions, which would be theoretically sound, is illustrated in several numerical experiments. Even in simple settings, elicitability however often fails, indicating the necessity to find validation procedures with non-zero elicitability breakdown point.