quantile interval score

[rob-taggart]

A couple of issues with scores.continuous.quantile_interval_score

I tried using preserve_dims='all' and got the following error message:

TypeError: Supplied dimensions 'all' must be an iterable of strings, not a string itself.

That makes me thing that quantile_interval_score is not using current way scores should be handling dimensions.

Then I went to read the API for the metric but couldn't find it. There is a tutorial, but I could not find quantile_interval_score in the list of metrics under scores.continuous in the API.

cc to @reza-armuei , I think you may implemented this one? Apologies if not.

[tennlee]

Thanks for noting this Rob. I'll take a look tomorrow and we can see what needs doing.

[tennlee]

I do see https://scores.readthedocs.io/en/latest/api.html#scores.continuous.quantile_interval_score in the docs currently. Is that what you were looking for in terms of API documentation?

[reza-armuei]

Thanks Rob for raising the issue about use of preserve_dims='all' in quantile_interval_score. I just checked the code and I noticed that it is not an issue with handling the dimension in quantile_interval_score itself, but rather it relates to check_dims function that is not designed to handle this option. I believe it is a simple fix in quantile_interval_score - if we first apply gather_dimensions function and use the returned dims in check_dims we should not face any issue (I did a test and it worked for me). So I will fix this issue by tomorrow.

[Steph-Chong]

@rob-taggart I am not sure whether you were looking for the quantile_interval_score API entry in the readthedocs documentation or in the code.

But I was reflecting on the ease of navigability of the API docs as rendered in readthedocs.

In the "included" table - https://scores.readthedocs.io/en/stable/included.html - within each section, we have listed each metric in alphabetical order.

However, in the API docs in readthedocs - https://scores.readthedocs.io/en/stable/api.html - there is not a consistent approach to how the entries are ordered within each section. While some of the sections list the entries in alphabetical order, mostly this is not the case. Sometimes the metrics have been sort of "thematically" grouped together (e.g. listing pearsonr next to spearmanr).

When there were a smaller number of metrics in the package, the lack of a consistent approach may not have mattered. However, with the growing number of metrics, have the API docs as rendered in readthedocs become a little difficult to navigate?

Personally, I find the easiest way to find API entries in readthedocs is to either use "Control F", or to find it in the "included" table and follow the link from there. But if those approaches aren't jumping out as options to other people, then perhaps this is an insufficient approach.

Would it make sense to reorder the API entries in readthedocs so things are easier to find? If so, how should it be reordered? Alphabetically? Another approach?

[Steph-Chong]

@rob-taggart I was wondering if you had seen my comment above about the ease/difficulty of finding API entries in readthedocs?

Do you think we should make it easier for people to find individual API entries in readthedocs (https://scores.readthedocs.io/en/stable/api.html)?

Reordering the API entries in readthedocs should be straight forward. So if you think putting the API entries in a different order would make it easier to find things, I am happy to submit a pull request doing so.

If so, how should it be reordered? Alphabetically (e.g. within scores.continuous each metric listed alphabetically, withinscores.probability each metric listed alphabetically etc.)? Another approach?

If you don't think any changes should be made, and it should be left as is, no worries.