Training with Large Dataset Causes Infinite Stall

[fishbotics]

🐛 Bug

I am training training two jobs that use the exact same code but different h5py files (same keys) to represent the data. One of the hdf5 files leads to a dataset that has 650,000,000 data points in the training set and one has 150,000,000. The training job with the larger dataset seems to just stop doing anything after a number of hours. After some number of hours, the CPU activity seems to die down. Neither job is out of memory (either system or GPU) but the larger job just seems to... stop working (although it doesn't crash).

Any ideas what might be going on? Both are still running and I can provide any info that might be helpful.

PyTorch Lightning Version (e.g., 1.5.0): 1.6.3
PyTorch Version (e.g., 1.10): 1.11
Python version (e.g., 3.9): 3.8
OS (e.g., Linux): Ubuntu 20.04
CUDA/cuDNN version: 11.3
GPU models and configuration: 8 A100s using ddp
How you installed PyTorch (conda, pip, source): pip
If compiling from source, the output of torch.config.show():
Any other relevant information:

Additional context

cc @Borda @akihironitta

[fishbotics]

I know this isn't exactly helpful, but I finally killed the job and it seems like it's some threading race condition thing. When I kill the process, the traceback looks like all of the DDP processes were waiting on a lock

Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 318, in _bootstrap
    util._exit_function()
  File "/usr/lib/python3.8/multiprocessing/util.py", line 360, in _exit_function
    _run_finalizers()
  File "/usr/lib/python3.8/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/usr/lib/python3.8/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 195, in _finalize_join
    thread.join()
  File "/usr/lib/python3.8/threading.py", line 1011, in join
    self._wait_for_tstate_lock()
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 1027, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):

However, I can't seem to kill the whole job with ctrl-C. I have to actually send a kill signal. There must be some hanging process.

Unfortunately, I don't see anything else of interest in there. The GPU utilization was high on several GPUs (and 0 on others), but the memory seemed fine. I saw #5725, which looks similar to this issue but my job wasn't out of memory.

For what it's worth, I tried the fix mentioned in that other thread (move the metrics to cpu) but this does not seem to work with DDP. I get an error saying the tensors need to on CUDA and dense.

[GGGGGGXY]

same issue.
when GPU utilization is 100%, but the training is hanging. Need ctrl+c to kill the training and start again.
My datasets is about 180G and model paras is about 700M.

[carmocca]

Unfortunately, we would need more information and ideally a reproducible script to check what could be the issue.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[angadkalra]

I'm having the exact same issue. For me, it looks like with a large dataset (500 GBs, 3000 data points), the machine free memory gets very low and it's causing some sort of process hang. The machine has total memory 256 GBs, the buff/cache during training is 230gb, used is 19gb, and free is around 4gb but goes up and down a lot. The data files are stored in EFS and mounted on the machine (same region).

In the middle of training, training will freeze in the middle of an epoch and when I check whats going on with top i, theres only a few python processes left (5 when using num_workers = 4) and 3 out of 4 GPUs are at 100%, the other gpu is at 0%. I barely get to 20 epochs before I have to hit ctrl+z, kill all python processes, and resume training from last.ckpt. CTRL+C doesn't work most of the time to kill training.

When I run training with 1000 data points (160 GBs), theres 120gb free memory the entire time, buff/cache is at 105gb and used is around 20gb. Training never stops and runs at 2it/s the entire time. I'm able to run 150 epochs in a row.

[angadkalra]

@carmocca Why does the machine memory buff/cache increase with dataset size? What are we storing across all epochs?

[angadkalra]

@Borda can you please help with this?

[Borda]

@Borda can you please help with this?

could you share your code to reproduce?