Evaluating infectious disease forecasts in a cost-loss situation

Abstract: In order for epidemiological forecasts to be useful for decision-makers the forecasts need to be properly validated and evaluated. Although several metrics fore evaluation have been proposed and used none of them account for the potential costs and losses that the decision-maker faces. We have adapted a decision-theoretic framework to an epidemiological context which assigns a Value Score (VS) to each model by comparing the expected expense of the decision-maker when acting on the model forecast to the expected expense obtained from acting on historical event probabilities. The VS depends on the cost-loss ratio and a positive VS implies added value for the decision-maker whereas a negative VS means that historical event probabilities outperform the model forecasts. We apply this framework to a subset of model forecasts of influenza peak intensity from the FluSight Challenge and show that most models exhibit a positive VS for some range of cost-loss ratios. However, there is no clear relationship between the VS and the original ranking of the model forecasts obtained using a modified log score. This is in part explained by the fact that the VS is sensitive to over- vs. under-prediction, which is not the case for standard evaluation metrics. We believe that this type of context-sensitive evaluation will lead to improved utilisation of epidemiological forecasts by decision-makers.