Really useful work! However, I encountered the following issue while running the program:
(grootv) hqu@hqu-System-Product-Name:~/user/MambaTree-main$ bash GrootV/scripts/bash_cls_train.sh --data-path /home/hqu/2tb/datasets/imagenet/imagenet
/home/hqu/anaconda3/envs/grootv/lib/python3.9/site-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
warnings.warn(
=> merge config from GrootV/classification/config/grootv_t_1k_224.yaml
RANK and WORLD_SIZE in environ: 0/1
Traceback (most recent call last):
File "/home/hqu/user/MambaTree-main/GrootV/classification/main.py", line 627, in <module>
torch.distributed.barrier()
File "/home/hqu/anaconda3/envs/grootv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3145, in barrier
work = default_pg.barrier(opts=opts)
RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1201, internal error, NCCL version 2.14.3
ncclInternalError: Internal check failed.
Last error:
Bootstrap : no socket interface found
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2753148) of binary: /home/hqu/anaconda3/envs/grootv/bin/python3.9
Traceback (most recent call last):
File "/home/hqu/anaconda3/envs/grootv/bin/torchrun", line 7, in <module>
sys.exit(main())
File "/home/hqu/anaconda3/envs/grootv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/home/hqu/anaconda3/envs/grootv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
run(args)
File "/home/hqu/anaconda3/envs/grootv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
elastic_launch(
File "/home/hqu/anaconda3/envs/grootv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/hqu/anaconda3/envs/grootv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
GrootV/classification/main.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-09-10_23:27:58
host : hqu-System-Product-Name
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 2753148)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Is it possible that I missed specifying some parameters in my command line instruction? I haven't worked with these parameters before, so I’m not very clear on their purpose.
Really useful work! However, I encountered the following issue while running the program:
Is it possible that I missed specifying some parameters in my command line instruction? I haven't worked with these parameters before, so I’m not very clear on their purpose.