Request: Automatic Worker Switching with Timeout Parameter for Data Loading

[Heccc257]

Type : Feature Request / Question

Description :
During training, data loading workers may occasionally hang (e.g., due to temporarily unavailable data files). This blocks the training pipeline until the worker recovers. To improve fault tolerance, I propose adding a timeout parameter to automatically switch to another worker if a worker fails to load data within the specified time.

Example Scenario :

Worker 1 starts loading data but hangs due to I/O issues.
After timeout=30s, the system should terminate Worker 1 and assign the task to Worker 2.
Suggested Implementation :

Add a timeout parameter to the DataLoader configuration.
Use a watchdog mechanism to monitor worker activity and reassign tasks on timeout.

[philipp-fischer]

Hi, with our current architecture, this would be very complicated to implement. Also note that we split the overall data across workers. If another worker continues, this changes the order of the data, which we cannot do.

So our requirement to always yield the same data order already contradicts the "skip if I/O problem" paradigm, since the other worker would likely also have an I/O issue with that data sample.

Note though that we do have a new watchdog feature which will at least tell you if a worker hangs and where they are hanging.