Do we have any plans on supporting pipeline parallel?

[LSX-Sneakerprogrammer]

Hi, because recently I'd like to fine-tune bloom-7b1 by ds-chat using full model parameters, while I find it does not have any supports on pipeline parallel. Do we have any plans on supporting pipeline parallel in the future?

[awan-10]

Bloom-7B can be finetuned by either tensor parallelism or zero-stage-2. Is there a particular reason for doing pipeline parallel?

[LSX-Sneakerprogrammer]

Bloom-7B can be finetuned by either tensor parallelism or zero-stage-2. Is there a particular reason for doing pipeline parallel?

Hi, if I understand correctly, tensor parallelism can only be used in the inference phase of PPO. When I use zero-2 to train bloom-7b without lora during SFT stage, I can not run the process with max_length > 256 using 8 A100-40G gpus. Thus, I consider pipeline parallelism will be helpful if supported.