-
Notifications
You must be signed in to change notification settings - Fork 179
Description
Is your feature request related to a problem? Please describe.
Currently, when evaluating the PinballLoss metric with an alpha parameter that is missing from the predictions (y_pred), it raises a generic ValueError:
"not all quantile values in alpha are available in y_pred"
There is an active TODO comment in skpro/metrics/_classes.py acknowledging this:
# todo: make error msg more informative - which alphas are missing
This can be frustrating for users as they have to manually debug which expected quantiles are missing from their forecasted DataFrame.
Describe the solution you'd like
I'd like the error message to explicitly list the specific alpha values that are missing. The PinballLoss._evaluate_by_index method should compute the set difference between the requested alpha and the provided y_pred_alphas and include them in the exception.
- Old Error Message:
ValueError: "not all quantile values in alpha are available in y_pred" - New Error Message:
ValueError: "not all quantile values in alpha are available in y_pred. Missing alphas: [0.1, 0.9]"
Describe alternatives you've considered
Leaving the error message as-is was considered, but calculating the missing values using set(alpha) - set(y_pred_alphas) has minimal overhead and significantly improves the developer experience when debugging probabilistic pipelines.
Additional context
I have already implemented this fix locally and updated test_evaluate_alpha_negative in test_probabilistic_metrics.py to verify the new error message format using pytest.raises. I am ready to submit a PR for this!