Error when using 3 GPUs

[JiSuanJiDaWang]

Hi, I encountered the following error when using 3 gpu to train smoke. However, when using 2 GPUs, it can run.

WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 10216 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 10217 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 0 (pid: 10215) of binary: /home/sysadmin/anaconda3/envs/mmd1.1x/bin/python
Traceback (most recent call last):
  File "/home/sysadmin/anaconda3/envs/mmd1.1x/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/sysadmin/anaconda3/envs/mmd1.1x/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/sysadmin/anaconda3/envs/mmd1.1x/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/home/sysadmin/anaconda3/envs/mmd1.1x/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/home/sysadmin/anaconda3/envs/mmd1.1x/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/home/sysadmin/anaconda3/envs/mmd1.1x/lib/python3.8/site-packages/torch/distributed/run.py", line 710, in run
    elastic_launch(
  File "/home/sysadmin/anaconda3/envs/mmd1.1x/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/sysadmin/anaconda3/envs/mmd1.1x/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
./tools/train.py FAILED
------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2022-12-06_18:39:13
  host      : WK-AISVR01
  rank      : 0 (local_rank: 0)
  exitcode  : -9 (pid: 10215)
  error_file: <N/A>
  traceback : Signal 9 (SIGKILL) received by PID 10215
^[[A^[[A^C[1]+  Exit 1                  CUDA_VISIBLE_DEVICES=1,2,3 PORT=29502 nohup ./tools/dist_train.sh configs/smoke/smoke-for-waymo.py 3