t2i finetune OOM problem

[godwenbin]

i have read your blog:https://thudm.github.io/CogKit/Finetune/Prerequisites/, but when i use your script train_ddp_t2i.sh, and i always OOM, my device is A100 80G. single a100 or multi-gpu all OOM.

[godwenbin]

i try to full finetune cogview4-6B

[OleehyO]

For full fine-tuning of cogview4-6B, you must use ZeRO distributed instead of DDP, so you need to refer to the train_zero_t2i.sh script.

[godwenbin]

@OleehyO i have used ZeRO distributed trainning with the script train_zero_t2i.sh script. And i can train with your demo 5 pic, but when try to use my own data,6k pics, it always meets watchdog ncll bug,can you tell me how to fix this bug?
Training steps: 0%| | 0/3520 [00:00<?, ?it/s][2025-04-08 07:20:15,767] [WARNING] [lr_schedules.py:683:get_lr] Attempting to get learning rate from scheduler before it has started
04/08/2025 07:20:16 - WARNING - DiffusionTrainer - Image object does not have filename attribute, skipping caching.
04/08/2025 07:20:16 - WARNING - DiffusionTrainer - Image object does not have filename attribute, skipping caching.
[rank0]:[E408 07:30:17.905391033 ProcessGroupNCCL.cpp:616] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=409, OpType=ALLREDUCE, NumelIn=457625664, NumelOut=457625664, Timeout(ms)=600000) ran for 600006 milliseconds before timing out.
[rank0]:[E408 07:30:17.905550088 ProcessGroupNCCL.cpp:1785] [PG ID 1 PG GUID 1 Rank 0] Exception (either an error or timeout) detected by watchdog at work: 409, last enqueued NCCL work: 410, last completed NCCL work: 408.
[rank1]:[E408 07:30:18.099917032 ProcessGroupNCCL.cpp:616] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=409, OpType=ALLREDUCE, NumelIn=457625664, NumelOut=457625664, Timeout(ms)=600000) ran for 600076 milliseconds before timing out.
[rank1]:[E408 07:30:18.100098252 ProcessGroupNCCL.cpp:1785] [PG ID 1 PG GUID 1 Rank 1] Exception (either an error or timeout) detected by watchdog at work: 409, last enqueued NCCL work: 410, last completed NCCL work: 408.
[rank0]:[E408 07:30:18.786668044 ProcessGroupNCCL.cpp:1834] [PG ID 1 PG GUID 1 Rank 0] Timeout at NCCL work: 409, last enqueued NCCL work: 410, last completed NCCL work: 408.
[rank0]:[E408 07:30:18.786692068 ProcessGroupNCCL.cpp:630] [Rank 0] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank0]:[E408 07:30:18.786710576 ProcessGroupNCCL.cpp:636] [Rank 0] To avoid data inconsistency, we are taking the entire process down.
[rank0]:[E408 07:30:18.788341797 ProcessGroupNCCL.cpp:1595] [PG ID 1 PG GUID 1 Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=409, OpType=ALLREDUCE, NumelIn=457625664, NumelOut=457625664, Timeout(ms)=600000) ran for 600006 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fcaed0e3446 in /usr/local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7fcaee3f6772 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7fcaee3fdbb3 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fcaee3ff61d in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fcb36d8c5c0 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x8609 (0x7fcb3bcf3609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7fcb3babe133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 1 PG GUID 1 Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=409, OpType=ALLREDUCE, NumelIn=457625664, NumelOut=457625664, Timeout(ms)=600000) ran for 600006 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fcaed0e3446 in /usr/local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7fcaee3f6772 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7fcaee3fdbb3 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fcaee3ff61d in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fcb36d8c5c0 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x8609 (0x7fcb3bcf3609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7fcb3babe133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fcaed0e3446 in /usr/local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: + 0xe4271b (0x7fcaee06c71b in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x145c0 (0x7fcb36d8c5c0 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #3: + 0x8609 (0x7fcb3bcf3609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x43 (0x7fcb3babe133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

[rank1]:[E408 07:30:18.793243520 ProcessGroupNCCL.cpp:1834] [PG ID 1 PG GUID 1 Rank 1] Timeout at NCCL work: 409, last enqueued NCCL work: 410, last completed NCCL work: 408.
[rank1]:[E408 07:30:18.793270154 ProcessGroupNCCL.cpp:630] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank1]:[E408 07:30:18.793278188 ProcessGroupNCCL.cpp:636] [Rank 1] To avoid data inconsistency, we are taking the entire process down.
[rank1]:[E408 07:30:18.795051747 ProcessGroupNCCL.cpp:1595] [PG ID 1 PG GUID 1 Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=409, OpType=ALLREDUCE, NumelIn=457625664, NumelOut=457625664, Timeout(ms)=600000) ran for 600076 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fb44b6cd446 in /usr/local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7fb44c9e0772 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7fb44c9e7bb3 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fb44c9e961d in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fb4953765c0 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x8609 (0x7fb49a2dd609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7fb49a0a8133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 1 PG GUID 1 Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=409, OpType=ALLREDUCE, NumelIn=457625664, NumelOut=457625664, Timeout(ms)=600000) ran for 600076 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fb44b6cd446 in /usr/local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7fb44c9e0772 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7fb44c9e7bb3 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fb44c9e961d in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fb4953765c0 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x8609 (0x7fb49a2dd609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7fb49a0a8133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fb44b6cd446 in /usr/local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: + 0xe4271b (0x7fb44c65671b in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x145c0 (0x7fb4953765c0 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #3: + 0x8609 (0x7fb49a2dd609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x43 (0x7fb49a0a8133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

[rank4]:[E408 07:30:19.155882841 ProcessGroupNCCL.cpp:616] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=409, OpType=ALLREDUCE, NumelIn=457625664, NumelOut=457625664, Timeout(ms)=600000) ran for 600040 milliseconds before timing out.
[rank4]:[E408 07:30:19.156094164 ProcessGroupNCCL.cpp:1785] [PG ID 1 PG GUID 1 Rank 4] Exception (either an error or timeout) detected by watchdog at work: 409, last enqueued NCCL work: 410, last completed NCCL work: 408.
[rank6]:[E408 07:30:19.331637786 ProcessGroupNCCL.cpp:616] [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=409, OpType=ALLREDUCE, NumelIn=457625664, NumelOut=457625664, Timeout(ms)=600000) ran for 600077 milliseconds before timing out.
[rank6]:[E408 07:30:19.331796282 ProcessGroupNCCL.cpp:1785] [PG ID 1 PG GUID 1 Rank 6] Exception (either an error or timeout) detected by watchdog at work: 409, last enqueued NCCL work: 410, last completed NCCL work: 408.
[rank4]:[E408 07:30:20.529232919 ProcessGroupNCCL.cpp:1834] [PG ID 1 PG GUID 1 Rank 4] Timeout at NCCL work: 409, last enqueued NCCL work: 410, last completed NCCL work: 408.
[rank4]:[E408 07:30:20.529257540 ProcessGroupNCCL.cpp:630] [Rank 4] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank4]:[E408 07:30:20.529266012 ProcessGroupNCCL.cpp:636] [Rank 4] To avoid data inconsistency, we are taking the entire process down.
[rank4]:[E408 07:30:20.530912840 ProcessGroupNCCL.cpp:1595] [PG ID 1 PG GUID 1 Rank 4] Process group watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=409, OpType=ALLREDUCE, NumelIn=457625664, NumelOut=457625664, Timeout(ms)=600000) ran for 600040 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fc5a9d4e446 in /usr/local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7fc5ab061772 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7fc5ab068bb3 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fc5ab06a61d in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fc5f39f75c0 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x8609 (0x7fc5f895e609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7fc5f8729133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 1 PG GUID 1 Rank 4] Process group watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=409, OpType=ALLREDUCE, NumelIn=457625664, NumelOut=457625664, Timeout(ms)=600000) ran for 600040 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fc5a9d4e446 in /usr/local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7fc5ab061772 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7fc5ab068bb3 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fc5ab06a61d in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fc5f39f75c0 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x8609 (0x7fc5f895e609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7fc5f8729133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fc5a9d4e446 in /usr/local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: + 0xe4271b (0x7fc5aacd771b in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x145c0 (0x7fc5f39f75c0 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #3: + 0x8609 (0x7fc5f895e609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x43 (0x7fc5f8729133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

[rank6]:[E408 07:30:20.599401301 ProcessGroupNCCL.cpp:1834] [PG ID 1 PG GUID 1 Rank 6] Timeout at NCCL work: 409, last enqueued NCCL work: 410, last completed NCCL work: 408.
[rank6]:[E408 07:30:20.599433592 ProcessGroupNCCL.cpp:630] [Rank 6] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank6]:[E408 07:30:20.599442346 ProcessGroupNCCL.cpp:636] [Rank 6] To avoid data inconsistency, we are taking the entire process down.
[rank6]:[E408 07:30:20.600983377 ProcessGroupNCCL.cpp:1595] [PG ID 1 PG GUID 1 Rank 6] Process group watchdog thread terminated with exception: [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=409, OpType=ALLREDUCE, NumelIn=457625664, NumelOut=457625664, Timeout(ms)=600000) ran for 600077 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fc092d97446 in /usr/local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7fc0940aa772 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7fc0940b1bb3 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fc0940b361d in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fc0dca405c0 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x8609 (0x7fc0e19a7609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7fc0e1772133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 1 PG GUID 1 Rank 6] Process group watchdog thread terminated with exception: [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=409, OpType=ALLREDUCE, NumelIn=457625664, NumelOut=457625664, Timeout(ms)=600000) ran for 600077 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fc092d97446 in /usr/local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7fc0940aa772 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7fc0940b1bb3 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fc0940b361d in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fc0dca405c0 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x8609 (0x7fc0e19a7609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7fc0e1772133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fc092d97446 in /usr/local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: + 0xe4271b (0x7fc093d2071b in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x145c0 (0x7fc0dca405c0 in /usr/local/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #3: + 0x8609 (0x7fc0e19a7609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x43 (0x7fc0e1772133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

W0408 07:32:44.472000 16999 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 17207 closing signal SIGTERM
W0408 07:32:44.473000 16999 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 17208 closing signal SIGTERM
W0408 07:32:44.474000 16999 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 17209 closing signal SIGTERM
W0408 07:32:44.474000 16999 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 17210 closing signal SIGTERM
W0408 07:32:44.474000 16999 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 17211 closing signal SIGTERM
W0408 07:32:44.475000 16999 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 17212 closing signal SIGTERM
W0408 07:32:44.475000 16999 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 17213 closing signal SIGTERM
E0408 07:33:02.928000 16999 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: -6) local_rank: 0 (pid: 17206) of binary: /usr/local/bin/python3.10
Traceback (most recent call last):
File "/usr/local/bin/accelerate", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/usr/local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1153, in launch_command
deepspeed_launcher(args)
File "/usr/local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 846, in deepspeed_launcher
distrib_run.run(args)
File "/usr/local/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
elastic_launch(
File "/usr/local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
train.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time : 2025-04-08_07:32:44
host : tj-1014376-cogview4-6b-full-250408-master-0
rank : 0 (local_rank: 0)
exitcode : -6 (pid: 17206)
error_file: <N/A>
traceback : Signal 6 (SIGABRT) received by PID 17206

[godwenbin]

@OleehyO

[OleehyO]

Are you using a single node or multiple nodes?

[godwenbin]

@OleehyO you means one gpu or multi-gpu? This error only occurs in multi-gpu training, in train_zero_t2i.sh.

[OleehyO]

04/08/2025 07:20:16 - WARNING - DiffusionTrainer - Image object does not have filename attribute, skipping caching.
04/08/2025 07:20:16 - WARNING - DiffusionTrainer - Image object does not have filename attribute, skipping caching.

Alright, I noticed there are two warnings here, indicating that some images in your dataset lack the 'filename' attribute (usually because certain metadata is missing, preventing them from being parsed into PIL objects like JPG or PNG). It's recommended that you first write a script to reload the images and save them unchanged as .png files to ensure no metadata is missing.

If the error persists even after resolving these two warnings, it's advisable to further check whether torch.distributed is functioning properly (e.g., inter-GPU communication), as the subsequent logs appear to be related to this issue.