Diagnostic for comparing distributions

[jgonzalezab]

For precipitation, it is important to compare distributions, particularly to assess the tails (extreme precipitation). We could include several diagnostics designed for this purpose (e.g., skill score [1] or Wasserstein distance [2]). However, one of the main concerns is how sensitive these diagnostics are to the parts of the distribution we are most interested in (extremes).

An alternative approach is to directly plot the frequency distributions using a log vertical axis (e.g., Fig. 3a [3] or Fig. 5 [4]). The main drawback is that, given the number of models being compared, such a plot could become cluttered. This raises the question: how can we display all these models in a single plot? Or, alternatively, how could we split them into multiple plots? The latter would require some classification of the models being intercompared, which is not straightforward.

Another option would be to implement one or more of the aforementioned diagnostics and examine how well their results correspond to what we observe in the plots. If the diagnostics are consistent with what we see in the plots, we could rely on them, otherwise, we may need to show the findings directly in the plots.

Refs

[1] https://journals.ametsoc.org/view/journals/clim/20/17/jcli4253.1.xml
[2] https://docs.scipy.org/doc/scipy-1.11.4/reference/generated/scipy.stats.wasserstein_distance.html
[3] https://arxiv.org/abs/2407.14158
[4] https://journals.ametsoc.org/view/journals/aies/4/4/AIES-D-24-0121.1.xml

[jgonzalezab]

Another approach is to use one of the previously mentioned techniques (e.g., the Wasserstein distance) as a diagnostic for the entire PDF precipitation distribution, while employing a different method specifically designed to evaluate the tail of the distribution. One example of the latter could be the Frequency Bias Index (see, for instance, Figure 3 in [1]). The purpose of this diagnostic is to quantify the extent to which the model over- or under-predicts events above a certain threshold. Since our focus is on the tail of the distribution, we could use relatively high thresholds.

Focusing on Figure 3 in [1], we can see that this diagnostic could be summarized as a single number per model, which is a desirable feature for inclusion in the benchmark.

[1] https://egusphere.copernicus.org/preprints/2025/egusphere-2025-4716/

[jgonzalezab]

This example shows the application of the Frequency Bias Index (FBI) to the predictions of a DL model. In this case, the FBI is computed across several thresholds that represent the tail of the distribution. This diagnostic helps us evaluate, in this case, how the model underestimates these extreme events. Furthermore, by measuring the deviation from the perfect score (represented by the line at 1.0), we can summarize the diagnostic with a single value (0.566 in this case).

FBI.pdf

See #5