Training Seems to Run Indefinitely When limit_train_batches is Set to 1.0

[xubin04]

Description:When the parameter limit_train_batches is set to 1.0 to train the entire dataset, the training process appears to run indefinitely.This issue consistently occurs across different datasets and models. Additionally, when the limit_train_batches parameter is set to a value greater than the number of batches in the dataset, the training process runs for that many times.

[imJiawen]

Hi, @xubin04

This behavior is due to the data loading mechanism used in ProbTS. Specifically, for training data, we follow GluonTS’s setup and use gluonts.itertools.Cyclic, which treats the dataset as an infinite stream and does not reset at the start of each epoch.

As a result:

• When limit_train_batches=1.0, PyTorch Lightning tries to iterate through the entire (infinite) dataset, causing the training to run indefinitely.
• When limit_train_batches is set to a value larger than the actual number of batches, training runs for that many (excessive) steps, since the stream is infinite.

As a workaround, you can:

1. Manually set limit_train_batches to match your dataset size.
2. Modify TransformedIterableDataset in probts.data.datasets.single_horizon_datasets to use a different iterator (e.g., non-cyclic ones from gluonts.itertools).

We’ll consider adding more configurable data iteration options in a future update. Thanks!

[xubin04]

你好

此行为是由于 ProbTS 中使用的数据加载机制造成的。具体来说，对于训练数据，我们遵循 GluonTS 的设置并使用 gluonts.itertools.Cyclic，它将数据集视为无限流，并且不会在每个 epoch 开始时重置。

结果：

• 当 limit_train_batches=1.0 时，PyTorch Lightning 会尝试遍历整个（无限）数据集，从而导致训练无限期运行。
• 当 limit_train_batches 设置为大于实际批次数的值时，由于流是无限的，因此将运行该数量（过多）步骤的训练。

解决方法是：

1. 手动设置 limit_train_batches 以匹配您的数据集大小。
2. 在 probts.data.datasets.single_horizon_datasets 中修改 TransformedIterableDataset 以使用不同的迭代器（例如，来自 gluonts.itertools 的非循环迭代器）。

我们将考虑在未来的更新中添加更多可配置的数据迭代选项。谢谢！

I also found that when using csdi, the validation progress remains stagnant even when I set a small number of validation batches.

[xubin04]

你好

此行为是由于 ProbTS 中使用的数据加载机制造成的。具体来说，对于训练数据，我们遵循 GluonTS 的设置并使用 gluonts.itertools.Cyclic，它将数据集视为无限流，并且不会在每个 epoch 开始时重置。

结果：

• 当 limit_train_batches=1.0 时，PyTorch Lightning 会尝试遍历整个（无限）数据集，从而导致训练无限期运行。
• 当 limit_train_batches 设置为大于实际批次数的值时，由于流是无限的，因此将运行该数量（过多）步骤的训练。

解决方法是：

1. 手动设置 limit_train_batches 以匹配您的数据集大小。
2. 在 probts.data.datasets.single_horizon_datasets 中修改 TransformedIterableDataset 以使用不同的迭代器（例如，来自 gluonts.itertools 的非循环迭代器）。

我们将考虑在未来的更新中添加更多可配置的数据迭代选项。谢谢！

If the training set has certain problems, will the validation and test sets also have the same problems? I haven't observed this.

[imJiawen]

Hi @xubin04 ,

The seemingly stagnant validation progress in CSDI is due to its nature as a non-autoregressive diffusion model. It requires significant memory during inference, and when the number of variables is large, it processes them in mini-batches over the variable dimension. This makes inference slow.

Note that the validation and test sets will not be affected by limit_train_batches. In TransformedIterableDataset, only the training set uses Cyclic:

self.transformed_dataset = TransformedDataset(
    Cyclic(dataset) if is_train else dataset,
    transform,
    is_train=is_train,
)

The validation and test sets are pre-split using the get_rolling_test function to ensure fair evaluation. This means that for each dataset and forecasting horizon, the number of validation/testing instances is fixed. A larger test_batch_size results in fewer testing iterations but higher memory usage; conversely, a smaller test_batch_size reduces memory usage but requires more iterations to complete.

For more details on preprocessing, please refer to the data_manager.