Towards Fair Comparisons of AI- and Physics-Based Weather Models for Extreme Events via the Weighted Potential CRPS

Abstract: We study whether deterministic AI weather prediction (AIWP) models issue more informative forecasts for extreme weather events than deterministic numerical weather prediction (NWP) models. The deterministic model output is subjected to statistical post-processing via isotonic distributional regression (IDR), or EasyUQ, before the resulting probabilistic forecasts are assessed using weighted versions of the continuous ranked probability score (CRPS). This extends the Potential CRPS (PCRPS) measure proposed by Gneiting et al. (2026) to focus on extreme outcomes. Since IDR exhibits optimality properties with respect to weighted versions of the CRPS, the proposed approach inherits desirable properties of the PCRPS, and, in particular, facilitates fair comparisons between data-driven and physics-based models when forecasting extreme weather events. We apply this evaluation framework to forecasts in the WeatherBench 2 dataset issued by the AIWP models GraphCast, Pangu-Weather, and FuXi, with the ECMWF's high-resolution NWP model serving as a physics-based reference. The forecast models are compared when predicting mean sea level pressure, temperature, wind speed, and precipitation extremes, defined as exceedances or non-exceedances of thresholds obtained from historical observation data. We additionally study forecast performance when predicting record-breaking events, though the ordering of the different methods is largely insensitive to the thresholds on which emphasis is placed. We find that AIWP models, particularly FuXi, result in the most informative forecasts for extreme weather events across most settings, suggesting that AIWP models have the potential to outperform NWP models when forecasting extremes.

• The paper proposes a weighted potential CRPS framework to convert deterministic forecasts into probabilistic ones for extreme event prediction.
• It employs EasyUQ with isotonic regression to transform AI and NWP forecasts, demonstrating that AI models like FuXi outperform traditional systems across multiple variables.
• The study validates twPCRPS as a robust proxy for forecast skill, encouraging further development of AI-driven probabilistic methods for extreme weather.

Fair Benchmarking of AI and Physics-Based Weather Models for Extremes via the Weighted Potential CRPS

Introduction

Assessment of AI-based weather prediction (AIWP) models relative to physics-based numerical weather prediction (NWP) systems for extremes remains non-trivial due to different training objectives and the challenge of fair evaluation. The study "Towards Fair Comparisons of AI- and Physics-Based Weather Models for Extreme Events via the Weighted Potential CRPS" (2606.21170) proposes a rigorous framework leveraging threshold-weighted versions of the potential continuous ranked probability score (twPCRPS) and isotonic distributional regression (EasyUQ) for post-processing deterministic forecasts into probabilistic form. This approach allows for theoretically sound and practically fair comparison of data-driven and physics-based models targeted specifically at extreme event forecasting.

Methodology

The methodology involves ensemble conversion of deterministic AIWP and NWP model outputs using EasyUQ, based on Isotonic Distributional Regression, to produce probabilistic forecasts. These are then verified with a suite of scoring rules, most notably threshold-weighted CRPS (twCRPS), which integrates a weight function to focus on user-specified regions of the outcome space (e.g., above the 99th percentile or record-breaking events).

For a deterministic forecast $x$, its corresponding potential twCRPS is defined as the twCRPS of its in-sample post-processed EasyUQ distribution. These scores can be interpreted as the potential informativeness of the deterministic backbone for probabilistic prediction of the specified extreme region. Crucially, EasyUQ predictive distributions are provably optimal with respect to twCRPS and quantile-weighted CRPS (qwCRPS) over the test set, ensuring that comparison does not introduce bias toward any model class.

Data and Model Evaluation Protocol

The comparison employs the WeatherBench 2 dataset, encompassing co-located global forecasts from ECMWF IFS-HRES (NWP/human-engineered), and three of the latest AIWP models: GraphCast, Pangu-Weather, and FuXi. Variables under analysis include 2 m temperature, 10 m wind speed, mean sea level pressure (MSLP), and 24-hour accumulated precipitation. Evaluation targets the 2020 calendar year, with historical quantiles for extremes (including records) derived from an independent ERA5 archive (1979-2019). Post-processing is performed gridpoint-wise, by variable and lead time, to ensure the clearest possible discrimination of model discriminatory skill with respect to extremes.

Main Results

Skill of Post-Processed Deterministic Models for Extremes

AIWP models, particularly FuXi, demonstrate systematically higher potential skill for extremes compared to NWP, as measured by threshold-weighted PCRPS skill (twPCRPS-S), across almost all variables and lead times. The superiority is consistent for both upper and lower tail targeting, and robust to the choice of threshold (e.g., 99th percentile, monthly or overall record values).

Figure 1: Threshold-weighted $\operatorname{twPCRPS-S}$ as a function of threshold quantile for all models and variables, aggregated over gridpoints and lead times.

At broad thresholds, the ranking of models remains stable, and at the most extreme quantiles, the discrepancy becomes more pronounced, most notably for temperature and MSLP at lead times beyond five days. For precipitation and wind speed, performance differences are more muted and spatially heterogeneous, but AIWP models still compare favorably.

Verification at Local and Extreme Scales

Spatially explicit skill maps reveal that AIWP models—most notably FuXi—exhibit maximal skill benefits in the extratropics, though the precise spatial pattern varies by variable and threshold.

Figure 2: $\operatorname{twPCRPS-S}$ of FuXi for threshold exceedances (99th percentile) by variable and lead time.

Analysis of record-breaking events (e.g., when observations exceed all available historical values) confirms that AIWP approaches surpass NWP in extracting useful information from their deterministic predictions for potential extreme event detection.

Figure 3: Average twPCRPS comparing model performance for monthly record extremes, by lead time.

Model Dominance and Statistical Significance

Map-based visualizations of the best-performing model by gridpoint corroborate the dominance of AIWP approaches in providing potentially most-informative forecast information for extremes.

Figure 4: Best-performing model at each gridpoint for twPCRPS when evaluating monthly record exceedances. Color encodes model identity.

Bootstrap-based significance analysis finds that while FuXi's superiority is statistically robust at longer leads and for temperature/MSLP, GraphCast can be competitive at short leads, and for precipitation, the distinction is less clear, reflecting the overall challenge of extreme precipitation forecasting.

Proxy for Probabilistic Forecast Skill

Strong correlations are found between the potential CRPS (twPCRPS) of deterministic models and the actual CRPS of their ensemble-based probabilistic counterparts.

Figure 5: Scatterplots of twCRPS (GenCast ensemble) vs twPCRPS (GraphCast deterministic), showing strong positive dependence, especially at longer leads.

This substantiates the validity of twPCRPS-based evaluation as a proxy for the direct assessment of probabilistic forecast skill for extremes, supporting the framework's utility in benchmarking deterministic model potential.

Implications and Future Perspectives

The analysis supports the interpretability of twPCRPS as a metric of forecast information content independent of calibration or sharpness, enabling fairer comparison among fundamentally diverse modeling approaches. The consistently higher twPCRPS skill of AIWP models for extremes—contradicting some prior results that favored physics-based NWP for records [see e.g., (2606.21170)]—suggests that AIWP methods, typically optimized for MSE/RMSE, intrinsically capture key informational features relevant for extreme prediction but may be under-utilized if assessed only in mean event or conditional-on-extreme regimes.

From an operational and theoretical standpoint, this result points toward increased value in developing AIWP-centric probabilistic forecasting post-processors, with applications not just in classic single-variable extremes, but also in the multivariate or compound event regime. Challenges remain for extending these techniques to spatiotemporally coherent extremes and joint distributions, given the univariate nature of EasyUQ. Further, the results highlight that operational AIWP variants, when initialized consistently with NWP analyses, retain their advantage (as shown in operational-mode experiments).

Looking ahead, progress in generating calibrated probabilistic ensembles from AIWP cores—via stochastic augmentation, generative backbone design, or advanced post-processing—could shift the paradigm of extreme weather forecasting, especially when combined with weighted scoring rule optimization, deep uncertainty quantification methods, and fair benchmark datasets. Advances are also required for robust tail calibration and for the grounding of these methods in more observationally rich (less reanalysis-dependent) ground truth.

Conclusion

The weighted potential CRPS-based benchmarking framework offers a rigorous, transparent, and fair paradigm for evaluating the predictive value of deterministic weather models for extremes. Under strict and reproducible conditions, current AIWP models meaningfully surpass state-of-the-art NWP in potential informativeness for extreme event prediction, with FuXi currently the standout performer for temperature and MSLP at longer forecast horizons. These findings motivate further emphasis on the development of AIWP-based probabilistic systems, with weighted scoring rule-centered learning/training and objective benchmarking across all quantiles and physically relevant regions of the forecast distribution.