Understanding TS Dataset Concepts

[jonbeckman]

I have been doubting my understanding of how to use datasets properly in gluonts. My dataset has 1631624 rows, which I split 70/30 for training and testing:

fractions = np.array([0.7, 0.3])
train_df, test_df = np.array_split(
            df, (fractions[:-1].cumsum() * len(df)).astype(int))

I can get decent RMSE numbers so I think this works well enough.

However, does the 70/30 split introduce training bias since it only includes the first 70% of the dataset. This may be fine since that's the principle behind time series forecasting (using previous history to forecast future behavior), but seems off no? Should the 70/30 split be interlaced throughout the entirety of the dataset?

With the above thought in mind, I started investigating better ways to split my dataset. In your tutorial notebook you say:

we split the DataFrame/DataFrames into training and test data. We can then use the training data to train the model and the test data for prediction. For training we will use the entire dataset up to last prediction_length entries. For testing we feed the entire dataset into make_evaluation_predictions, which automatically splits the last prediction_length entries for us and returns their predictions.

This confuses me further because if I follow this with my dataset, my training dataset would be 1631619 rows (1631624 total - 5 prediction_length). That leaves almost nothing left for the testing dataset.

The only way I can make this make sense is if I transform my 1631624 row dataframe into many series of size context_length + prediction_length. But I see no guidance on explaining this and how to approach it. Does gluonts handle this behind the scenes where it iterates over the entire dataset with incrementing windows of size context_length + prediction_length?

Any guidence would be greatly appreciated!