Add coverage probability

[jonasbhend]

Coverage probability, or just coverage for short, measures the fraction of time, the observed outcome lies in a specified prediction interval of the forecast distribution. Thereby, coverage probability provides a necessary, but not sufficient, condition for forecast reliability and is closely related to the PIT and rank histograms. Coverage probability is particularly interesting to assess for forecasts for which only limited quantiles of the forecast distribution are communicated. In these cases, evaluating reliability using standard tools and metrics such as PIT and rank histograms is less straight-forward.

Empirical coverage of prediction intervals has not been covered widely in the scientific literature. Applications of the metric can be found in Schulz and Lerch, 2022 among others.

[tennlee]

Hi Jonas! Thanks for raising the issue, and sorry for the short delay in my reply. I had a chat to someone else in the team and talked through the metric. We'd be very happy to include it. Are you proposing to put up the code yourself, or would you prefer for the team to put it on the backlog? Note, if you want us to include it on the backlog, it may be quite some time, as I don't think we have an immediate need for this metric and so there are a number of other things which would take priority. However, if you'd like to put up a pull request we will be able to prioritise the review process.

[jonasbhend]

Hi Tenessee. Thanks for clarifying. I will try to find the time to submit a PR, but likely won’t have time in the coming weeks. If you don’t mind having it in your backlog too, please add it.
Cheers
Jonas