Skip to content

[ENH] Make missing alpha error message in PinballLoss more informative. #977

@MOHITKOURAV01

Description

@MOHITKOURAV01

Is your feature request related to a problem? Please describe.

Currently, when evaluating the PinballLoss metric with an alpha parameter that is missing from the predictions (y_pred), it raises a generic ValueError:

"not all quantile values in alpha are available in y_pred"

There is an active TODO comment in skpro/metrics/_classes.py acknowledging this:
# todo: make error msg more informative - which alphas are missing

This can be frustrating for users as they have to manually debug which expected quantiles are missing from their forecasted DataFrame.

Describe the solution you'd like

I'd like the error message to explicitly list the specific alpha values that are missing. The PinballLoss._evaluate_by_index method should compute the set difference between the requested alpha and the provided y_pred_alphas and include them in the exception.

  • Old Error Message: ValueError: "not all quantile values in alpha are available in y_pred"
  • New Error Message: ValueError: "not all quantile values in alpha are available in y_pred. Missing alphas: [0.1, 0.9]"

Describe alternatives you've considered

Leaving the error message as-is was considered, but calculating the missing values using set(alpha) - set(y_pred_alphas) has minimal overhead and significantly improves the developer experience when debugging probabilistic pipelines.

Additional context

I have already implemented this fix locally and updated test_evaluate_alpha_negative in test_probabilistic_metrics.py to verify the new error message format using pytest.raises. I am ready to submit a PR for this!

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions