Long audio (1 hour length+ ) SFT using qwen3omini-30B-A3B using deepspeed and grouped_mm

[robi56]

Checklist / 检查清单

• I have searched existing issues, and this is a new feature request. / 我已经搜索过现有的 issues，确认这是一个新的 Feature Request。

Feature Request Description / Feature Request 描述

I checked it : https://github.com/modelscope/ms-swift/blob/main/examples/train/moe/qwen3_moe.sh

Swift Command

CUDA_VISIBLE_DEVICES=0,1 NPROC_PER_NODE=2 ENABLE_AUDIO_OUTPUT=0 swift sft \
  --model "$MODEL" \
  --use_hf true \
  --dataset "$TRAIN_JSONL" \
  --val_dataset "$VAL_JSONL" \
  --split_dataset_ratio 0.01 \
  --load_from_cache_file true \
  --deepspeed zero3 \
  --experts_impl grouped_mm \
  --tuner_type lora \
  --torch_dtype bfloat16 \
  --lora_rank 16 \
  --lora_alpha 32 \
  --lora_dropout 0 \
  --target_modules q_proj k_proj v_proj o_proj \
  --attn_impl sdpa \
  --freeze_vit true \
  --freeze_aligner true \
  --padding_free false \
  --num_train_epochs 1 \
  --per_device_train_batch_size 1 \
  --per_device_eval_batch_size 1 \
  --gradient_accumulation_steps 1 \
  --learning_rate 3e-5 \
  --gradient_checkpointing true \
  --eval_strategy steps \
  --eval_steps 50 \
  --save_steps 50 \
  --save_total_limit 2 \
  --logging_steps 5 \
  --max_length 65500 \
  --warmup_ratio 0.05 \
  --dataset_num_proc 4 \
  --dataloader_num_workers 4 \
  --output_dir "$OUTPUT_DIR" \
  "${RESUME_ARG[@]}"

Log:

NVIDIA-SMI

Issue

There is no progress after it

Pull Request / Pull Request 信息

No response

[taking-lying-flat]

For this attention-only LoRA setup on 2× H200, I would first remove --deepspeed zero3 and keep NPROC_PER_NODE=2 to use ordinary DDP. If you want to keep DeepSpeed, try --deepspeed zero2 instead.

With ENABLE_AUDIO_OUTPUT=0, the Thinker's BF16 weights are approximately 59 GiB, so the base weights fit on each H200 without parameter sharding. LoRA only trains a small subset of parameters, while ZeRO-3 adds parameter gathering and a more complicated loading path. I would reserve ZeRO-3 for a measured memory requirement here. This does not guarantee that a 65K-token training step fits: long-audio activations and other temporary tensors still need to be measured, and ZeRO-3 does not itself partition the sequence activations.

The last log in your screenshot, model_kwargs: ..., is printed immediately before the model-loading call. This points to model loading/initialization, before audio preprocessing or a training step.

One concrete suspect is CPU memory pressure during MoE checkpoint conversion. In Transformers 5.16.1's ZeRO-3 loader, each rank performs the expert-weight conversion on CPU before loading the partitioned parameters. For this model, the fused BF16 expert tensors alone amount to roughly 54 GiB per rank. CPU conversion, paging/I/O, or waiting for another rank could explain allocated GPU memory with little GPU activity. This is a hypothesis, not a confirmed diagnosis; your installed versions matter.

Suggested checks:

1. First change only the DeepSpeed setting as above and check whether model loading completes; keep ENABLE_AUDIO_OUTPUT=0 and grouped_mm initially.
2. Once loading works, establish a short-audio baseline (e.g. --max_length 4096) and increase duration while measuring peak memory.
3. To diagnose the original ZeRO-3 stall, please share the ms-swift, transformers, torch, and deepspeed versions, host RAM/swap usage, and stack dumps of both worker processes while stuck (e.g. py-spy dump --pid <worker_pid>). These should distinguish CPU weight conversion/I/O from a collective wait.

[robi56]

DDP Experiment:

@taking-lying-flat thank you for helping,,

For larger audio than 10 minutes are failed due to the high token increases. The memory overflow occurs .

Experiment : Status : Successful : Audio length 12 minutes upto, --max_length 28000

Configuration:
max_limit_seconds = 10.0 * 60.0 [Audio is truncated by 10 minutes]

CUDA_VISIBLE_DEVICES=0,1 NPROC_PER_NODE=2 ENABLE_AUDIO_OUTPUT=0 PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' swift sft \
  --model "$MODEL" \
  --use_hf true \
  --dataset "$TRAIN_JSONL" \
  --val_dataset "$VAL_JSONL" \
  --split_dataset_ratio 0.01 \
  --load_from_cache_file true \
  --tuner_type lora \
  --torch_dtype bfloat16 \
  --lora_rank 16 \
  --lora_alpha 32 \
  --lora_dropout 0 \
  --target_modules q_proj k_proj v_proj o_proj \
  --attn_impl sdpa  \
  --freeze_vit true \
  --freeze_aligner true \
  --padding_free false \
  --num_train_epochs 1 \
  --per_device_train_batch_size 1 \
  --per_device_eval_batch_size 1 \
  --gradient_accumulation_steps 1 \
  --learning_rate 3e-5 \
  --gradient_checkpointing true \
  --eval_strategy steps \
  --eval_steps 50 \
  --save_steps 50 \
  --save_total_limit 2 \
  --logging_steps 5 \
  --max_length 28000 \
  --warmup_ratio 0.05 \
  --dataset_num_proc 4 \
  --dataloader_num_workers 4 \
  --output_dir "$OUTPUT_DIR" \
  "${RESUME_ARG[@]}"

Server Log

NVIDIA-SMI:

Dependences :

torch: 2.11.0

deepspeed: 0.19.7

triton: 3.6.0

transformers: 5.12.1

ms_swift: 4.4.1