the DeepSpeed-Chat demo train.py cannot even run

[Emerald01]

I have 8 A100 GPU setup at GCP, and if I run this demo, I got error like this

$$ /export/home/DeepSpeedExamples/applications/DeepSpeed-Chat# python train.py --actor-model facebook/opt-1.3b --reward-model facebook/opt-350m --deployment-type single_node

---=== Running Step 1 ===---
Running:
bash /export/home/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/training_scripts/single_node/run_1.3b.sh /export/home/DeepSpeedExamples/applications/DeepSpeed-Chat/output/actor-models/1.3b 
---=== Finished Step 1 in 0:00:09 ===---
---=== Running Step 2 ===---
Running:
bash /export/home/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step2_reward_model_finetuning/training_scripts/single_node/run_350m.sh /export/home/DeepSpeedExamples/applications/DeepSpeed-Chat/output/reward-models/350m 
---=== Finished Step 2 in 0:00:19 ===---
---=== Running Step 3 ===---
Traceback (most recent call last):
  File "train.py", line 210, in <module>
    main(args)
  File "train.py", line 194, in main
    cmd = get_cmd(args, step_num)
  File "train.py", line 160, in get_cmd
    verify_model(args, 1)  # Verify step 1 model exists
  File "train.py", line 149, in verify_model
    raise RuntimeError(error_str)
RuntimeError: Step 1 model has not been trained. Train it with:
python3 train.py --step 1 --actor-model 1.3b

So I guess it is not even running step 1, then we go to step 1 to run directly, based on the tutorial
now the log shows this

Caught signal 7 (Bus error: nonexistent physical address)
==== backtrace (tid:   7626) ====
 0 0x0000000000014420 __funlockfile()  ???:0
 1 0x000000000018bb41 __nss_database_lookup()  ???:0
 2 0x000000000006929c ncclGroupEnd()  ???:0
 3 0x000000000005e36d ncclGroupEnd()  ???:0
 4 0x0000000000008609 start_thread()  ???:0
 5 0x000000000011f133 clone()  ???:0
=================================
[2023-04-25 22:25:44,284] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 6329
[2023-04-25 22:25:44,900] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 6330
[2023-04-25 22:25:45,556] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 6331
[2023-04-25 22:25:46,171] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 6332
[2023-04-25 22:25:46,171] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 6333
[2023-04-25 22:25:46,786] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 6334
[2023-04-25 22:25:47,361] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 6335
[2023-04-25 22:25:47,935] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 6336
[2023-04-25 22:25:48,470] [ERROR] [launch.py:434:sigkill_handler] ['/opt/conda/bin/python3.8', '-u', 'main.py', '--local_rank=7', '--data_path', 'Dahoas/rm-static', 'Dahoas/full-hh-rlhf', 'Dahoas/synthetic-instruct-gptj-pairwise', 'yitingxie/rlhf-reward-datasets', '--data_split', '2,4,4', '--model_name_or_path', 'facebook/opt-1.3b', '--per_device_train_batch_size', '8', '--per_device_eval_batch_size', '8', '--max_seq_len', '512', '--learning_rate', '9.65e-6', '--weight_decay', '0.', '--num_train_epochs', '16', '--gradient_accumulation_steps', '1', '--lr_scheduler_type', 'cosine', '--num_warmup_steps', '0', '--seed', '1234', '--zero_stage', '2', '--deepspeed', '--output_dir', './output'] exits with return code = -7
"training.log" 58L, 5376C                                                                                     

However, 8 A100 GPUs are there dangling doing nothing.

I am wondering if I have any issue configuring this (I successfully installed everything following the doc) and are those scripts at their working conditions? And if there is any unit or module tests to run against before doing any real experiments? I feel that this codebase is not yet ready for users to jump in if the demo cannot even run, or it cannot, at least give some insightful messages.

[mrwyattii]

Hi @Emerald01 and sorry you are experiencing problems with this script. This is certainly not how we intend for the demo to function. If there are problems with any of the steps, it should exit the script with an error indicating that:
https://github.com/microsoft/DeepSpeedExamples/blob/f4ad1d5721630185a9088565f9201929a8b1ffdf/applications/DeepSpeed-Chat/train.py#L174

Could you please try running just step 1 on a single GPU and share the log output (if there is an error)?
python3 train.py --actor-model facebook/opt-1.3b --reward-model facebook/opt-350m --deployment-type single_gpu --step 1

The Caught signal 7 (Bus error: nonexistent physical address) indicates to me that this could be a problem in your environment, such as your CUDA installation.

[Emerald01]

Really appreciate your help. When I run one single GPU, it gives me OOM

python3 train.py --actor-model facebook/opt-1.3b --reward-model facebook/opt-350m --deployment-type single_gpu --step 1

RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 39.59 GiB total capacity; 38.48 GiB already allocated; 261.19 MiB free; 38.65 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2023-04-27 01:29:52,079] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 10103
[2023-04-27 01:29:52,080] [ERROR] [launch.py:434:sigkill_handler] ['/opt/conda/bin/python3.8', '-u', 'main.py', '--local_rank=0', '--model_name_or_path', 'facebook/opt-1.3b', '--gradient_accumulation_steps', '8', '--lora_dim', '128', '--zero_stage', '0', '--deepspeed', '--output_dir', '/export/home/DeepSpeedExamples/applications/DeepSpeed-Chat/output/actor-models/1.3b'] exits with return code = 1

Shall I change the default the setup to something more efficient?

In this sense I think the GPU environment or CUDA looks okay, I have no idea why it reports error code 7.

I used the base image from Nvidia: nvcr.io/nvidia/pytorch:22.08-py3
which is a pretty robust image that I used it for many projects, for single GPU or a node.
But if you have any standardized image in your suggestion list, please let me know.

[mrwyattii]

I think the issue here is that the demo we share were intended to be run on A6000 / A100-80GB and you are using 40GB version of the A100. We can fix this by reducing the training batch size to reduce memory requirements. Typically, the train.py will catch the OOM error and suggest this in the error message here - but it seems that did not happen for you. I'll try to recreate that and get a fix on my side.

But to address the issue of your OOM error, please try the following:

• Using your text editor of choice, open the single GPU 1.3B step 1 execution script (applications/DeepSpeed-Chat/training/step1_supervised_finetuning/training_scripts/single_gpu/run_1.3b.sh)
• Add the following after main.py on line 18: --per_device_train_batch_size 8 --per_device_eval_batch_size 8
• Try running the single GPU script again (same command as I shared above)

If this worked, you can do a similar change for the multi-GPU (i.e., single-node) deployment.

Also, we are currently working on autotuning of these parameters (https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/DeepSpeed-Chat#-deepspeed-chats-roadmap-). This should alleviate the headache of needing to manually change batch sizes for different GPU setups!