Minor typo bug when checking normalization choice for DDP training

[huhuman]

Thanks for the work for supporting DDP training. I noticed a likely bug when checking normalization choice for DDP:

if self.opt.norm == "syncbatch":
    raise ValueError(f"For distributed training, opt.norm must be 'syncbatch' or 'inst', but got '{self.opt.norm}'. " "Please set --norm syncbatch for multi-GPU training.")

which can be found https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/2a7afba2895d52556dd5dfe07e8555ef657ced6f/models/base_model.py#L118C21-L119C193

A quick fix can be

if self.opt.norm != "syncbatch" and self.opt.norm != "instance":
    raise ValueError(f"For distributed training, opt.norm must be 'syncbatch' or 'instance', but got '{self.opt.norm}'. " "Please set --norm syncbatch for multi-GPU training.")

The codes work for me under multi-GPU DDP after this minor revision.

[siddharthsrivastava0501]

I tried the fix that you suggested but my process still hangs after the creation of the models and dataloaders and eventually times out. Could you share the command-line parameters that worked for you after torchrun --nproc_per_node 4 train.py ...?

[huhuman]

I tried the fix that you suggested but my process still hangs after the creation of the models and dataloaders and eventually times out. Could you share the command-line parameters that worked for you after torchrun --nproc_per_node 4 train.py ...?

My params are just as simple as torchrun --nproc_per_node=4 train.py --norm syncbatch

I was training with only 2xA40 at that time on a HPC node, and it works. I just ran another one with 4xA40, and it seems to run ok by now for several iterations. I haven't verified if the results make sense but at least training won't get stuck.

Not sure if other params matter, but the full command for my own experiment that works as mentioned is

torchrun --nproc_per_node=4 train.py --norm syncbatch
    --dataroot /path/to/dataset \
    --name ... \
    --checkpoints_dir /path/to/save/checkpoint \
    --model cycle_gan  \
    --dataset_mode unaligned \
    --direction AtoB \
    --save_epoch_freq 20 \
    --gan_mode wgangp \
    --lr_policy cosine \
    --lr 0.0002 \
    --n_epochs 30 \
    --n_epochs_decay 30 \
    --preprocess scale_width_and_crop \
    --load_size 512 --crop_size 512 \
    --batch_size 2 --num_threads 4 \
    --pool_size 100 \
    --lambda_A 5 --lambda_B 5 --lambda_identity 0.1 \
    --netD n_layers --n_layers_D 4

Environments:

python=3.11.13
torch=2.4.0
torchvision=0.19.0
cuda=12.4