(low priority) Review interval score (and perhaps others) for clarity on what's normalised vs what's a natural value

[tennlee]

There is a general confusion that could be possible when interpreting confidence intervals and quantiles. Consider a value that has a natural range between -100 and +100 (being, for example, a practical range for temperatures), which may be normalised (or standardised) and be considered instead in the normalised space.

The various thresholds and parameters to the scores being calculated could sometimes be either read as being in the natural space, or in the normalised/probability space. For example, consider https://scores.readthedocs.io/en/latest/api.html#scores.continuous.interval_score . It has gradually become clear to me that the score is in the natural range rather than normalised or probability range. q_u is the forecast value at the quantile level, and the under-prediction penalty is based on the natural range rather than the normalised range.

This may or may not be obvious to users of the score. It might be worth providing some clarifying information to the return statement, or or to the documentation. I may even have gotten this wrong. But I don't think it's immediately clear that the value (range) of the score will be sensitive to the domain (natural values) of the input rather than being expressed as a probability or deviation.

The API docs and tutorial could benefit from someone reading through and making sure that it's reasonably clear at all stages from this perspective.

[nikeethr]

I don't know if this reasoning aligns with the concerns you're raising, but anyway:

For a gaussian distribution (special case), the "normalised" space is continuous whereas the quantiles are discretised bins. While they are computed using sorting, the "natural" values still conform to the standard distribution in some way.

In the scenario that the distribution is approximated from reference data, the lowest 25% of the values IS in the scenario of the distribution modelled from the reference data $$\text{CDF}^{-1}(0.25)$$, since the count is the in-place approximation for the probability density.

For all other non-conformant distributions without a reference, its not clear I guess.

[rob-taggart]

I think it is the responsibility of the user to sufficiently understand the characteristics of the scores they are working with.

We should give references to accessible, accurate and well-communicated papers/presentations of each score.

[nikeethr]

I think it is the responsibility of the user to sufficiently understand the characteristics of the scores they are working with.

We should give references to accessible, accurate and well-communicated papers/presentations of each score.

I agree with this. It's hard for scores to enforce what underlying standardisation techniques are used for different literature or classify commonalities - read the papers is probably the best guard against ambiguity. It can try where possible, but unless there's a general protocol/consensus in literature it feels like its pretty context dependent.