Rolling predictions on a test set

[ir0nt0ad]

Hi all,

I would like to split my time series data into a train and a test set and train a DeepAREstimator on the training set. I would then like to perform sliding window predictions on the test set, e.g. with prediction_length of 4, I would like to be able to take first 4 datapoints t1-t4 from the test set, create predictions for datapoints t5-t8 and calculate the error, then move my window one step to the right, take datapoints t2-t5, predict t6-t9 and so on.

Could you please let me know where to look for an example? Is https://github.com/awslabs/gluon-ts/blob/master/examples/run_rolling_forecast_backtest.py relevant? Many thanks!

[juanlopezcode]

hey, have you found how to apply this rolling window? anyone could help? thanks

[ir0nt0ad]

Hey, no eventually I abandoned DeepAR altogether.

[dmitra79]

I think you can do that by specifying StepStrategy for the rolling dataset, ex:

dataset_rolled = generate_rolling_dataset(
    dataset=dataset,
    strategy=StepStrategy(
        prediction_length=4,
        step_size=1,
    ),
)

Then to evaluate you'd need to define predictor and evaluator and run:

agg_metrics,item_metrics =  backtest.backtest_metrics(dataset_rolled, predictor, evaluator)

[rurjais]

This thread is of my interest.

How to you define the predictor and evaluator in order to count with the sliding window?

Could you provide a toy example on how to perform it?

Furthermore, is there any official source of information I can search for in order to build such setup?

Kind Regards.
Ricardo

[dmitra79]

You can start by looking at tutorial examples: https://ts.gluon.ai/stable/tutorials/forecasting/quick_start_tutorial.html

[rurjais]

Thank you for your kind reference dmitra79. However, as I made some tests on DeepAR and started to make some guesses, I have some questions:

• Isn't DeepAR natively build as sliding window? Since, from what I understand, if I have a dataset with 100 values, and use 80 for training, what will be done natively by the model is to model a series of sliding windows of the same size for both context_window and prediction_window (first context_window=[x0,x1,x2,...x29] and the first prediction_window = [x30], then the second context window will be = [x1,x2,x3,...,x30] and the second prediction_window=[x31], then the third context window will be = [x2,x3,x4,...,x31] and the third preditction_window=[x32], up until I finish with context_window = [x50,x51,x52,...x79] and the last prediction_window=[x80]) At least, this is how I understand it.

• As I understand the implementation generate_rolling_dataset function, what that function will do is an "expanding sliding" window, where the first context_window will have only [x0,x1,x2,x3] and the model will learn to predict [x4,x5,x6,x7]; then the second leg will have [x0,x1,x2,x3,x4] and learn to predict [x5,x6,x7,x8], and so on until we exhaust the available training data. Is this far from what is actually happening?

Kind Regards
Ricardo