How to use tp or tp when I traning models from transformers?

[Mulbetty]

Hi, I am now tring to train the qwen2.5 models using deepspeed and transformers, I tried to using tp or pp by setting the ds_config.json like:
"tensor_parallel":{"autotp_size":4},
"pipeline_parallel": {
"pp_size": 64,
"microbatches": 4,
"enable_backward_allreduce": true
},

But I think it doesn't woking. Should I wrapped the model with PipelineModule ?

[leejianwoo-collab]

Recommended Complete Configuration
{
"train_batch_size": 256,
"train_micro_batch_size_per_gpu": 1,
"gradient_accumulation_steps": 1,
"optimizer": {
"type": "Adam",
"params": {
"lr": 1e-4
}
},
"fp16": {
"enabled": true
},
"zero_optimization": {
"stage": 1
},
"tensor_parallel": {
"tp_size": 4
},
"pipeline_parallel": {
"pp_size": 64,
"microbatches": 4
}
}
With this configuration, DeepSpeed will correctly enable TP and PP parallelism strategies for your QWEN2.5 model training.

[south-ocean]

thanks

[rockcute]

Hi, I was wondering whether DeepSpeed supports using tensor parallelism and pipeline parallelism at the same time. Any clarification would be greatly appreciated.