Validation

[manumacc]

How do we validate models?

Possible approaches:

1. We have data starting on February 4 2021 00:00:00 and up to February 24 2021 23:59:56. For a total of 21 days or 3 weeks. We can treat the data as a time series and hold out the last X days from the training set to use them as validation data.
2. Split train/test (X/Y%), grouping by tweet_id to keep all data points related to the same tweet in the same fold.
   a. Maybe it doesn't make sense to use values in the future to predict values in the past, so we probably shouldn't use a validation set from the past to validate a model trained on future samples.
   b. On the other hand, maybe tweets are (almost) stationary data, so they're not much affected by the shift in time -- also considering we're not talking about years, but it's a small one-month window so we can probably consider it stationary? -> See eda-stationarity.ipynb
3. Other interesting ideas on cross validation for time series data: https://stats.stackexchange.com/questions/14099/using-k-fold-cross-validation-for-time-series-model-selection