Large data support

[tangjinou]

how can this lib support the large data?

we use tft in lib to predict well in several mills data ,

but one day we need to support 4 times bigger data with tft

how can we solve it ? we find the out memory is limit， so can we extend the dataload?

[benHeid]

I assume that the dataset is currently a memory limit, since it keeps everything in memory. According to the documentation of the TimeSeriesDataset, you could try the following:

        Large datasets:

            Currently the class is limited to in-memory operations (that can be sped up by an
            existing installation of `numba <https://pypi.org/project/numba/>`_). If you have extremely large data,
            however, you can pass prefitted encoders and and scalers to it and a subset of sequences to the class to
            construct a valid dataset (plus, likely the EncoderNormalizer should be used to normalize targets).
            when fitting a network, you would then to create a custom DataLoader that rotates through the datasets.
            There is currently no in-built methods to do this.