Description
Current Problem
When evaluating forecasting models, commonly used point metrics (MAE/RMSE/MAPE/MASE, etc.) don’t capture how well predictions align with the target across multiple tolerance levels, which is often what practitioners judge visually (especially for operational forecasting dashboards).
Overlay-DX is designed to address this: it builds a curve of “% of predictions within tolerance” across decreasing tolerance thresholds, and reports a normalized AUC score in [0, 1] (higher is better). It tends to be less dominated by outliers than squared-error based metrics, while remaining interpretable
Desired Workflow
from gluonts.evaluation import Evaluator
from overlay_dx import overlay_dx_score # or whatever the public function is
def overlay_dx_gluonts(target, forecast):
# target, forecast are numpy arrays (forecast is mean/median depending on config)
return float(overlay_dx_score(target, forecast,
max_percentage=100.0,
min_percentage=0.1,
step=0.1))
evaluator = Evaluator(
custom_eval_fn={
"overlay_dx": [overlay_dx_gluonts, "mean", "median"]
}
)
agg_metrics, item_metrics = evaluator(ts_iterator, forecast_iterator)
print("Overlay-DX:", agg_metrics["overlay_dx"])
Use cases
-
Forecasting evaluation where visual alignment matters (stakeholder review, ops planning)
-
Scenarios where outlier sensitivity of MSE/RMSE is problematic
-
Understanding model quality at multiple tolerance levels, not just one error scalar
-
Model selection where “good enough within tolerance” is more meaningful than absolute error minimization
Proposed change
Add Overlay-DX as a built-in metric function in gluonts.evaluation.metrics
def overlay_dx(
target: np.ndarray,
forecast: np.ndarray,
*,
max_percentage: float = 100.0,
min_percentage: float = 0.1,
step: float = 0.1,
) -> float:
"""Overlay-DX metric (normalized AUC of tolerance-sweep coverage curve)."""
How it works (high level)
-
Define tolerance bands as percentages of the target range
-
For each tolerance level: compute the fraction of forecasts within the band
-
Compute AUC under the “coverage vs tolerance” curve
-
Normalize to produce a score in [0, 1] (higher is better)
References
Description
Current Problem
When evaluating forecasting models, commonly used point metrics (MAE/RMSE/MAPE/MASE, etc.) don’t capture how well predictions align with the target across multiple tolerance levels, which is often what practitioners judge visually (especially for operational forecasting dashboards).
Overlay-DX is designed to address this: it builds a curve of “% of predictions within tolerance” across decreasing tolerance thresholds, and reports a normalized AUC score in [0, 1] (higher is better). It tends to be less dominated by outliers than squared-error based metrics, while remaining interpretable
Desired Workflow
Use cases
Forecasting evaluation where visual alignment matters (stakeholder review, ops planning)
Scenarios where outlier sensitivity of MSE/RMSE is problematic
Understanding model quality at multiple tolerance levels, not just one error scalar
Model selection where “good enough within tolerance” is more meaningful than absolute error minimization
Proposed change
Add Overlay-DX as a built-in metric function in
gluonts.evaluation.metricsHow it works (high level)
Define tolerance bands as percentages of the target range
For each tolerance level: compute the fraction of forecasts within the band
Compute AUC under the “coverage vs tolerance” curve
Normalize to produce a score in [0, 1] (higher is better)
References