@@ -50,6 +50,44 @@ The model must output JSON of the form:
5050{"text" : " " , "videos" : [" /abs/path/to/generated.mp4" ]}
5151```
5252
53+ ### Full example (multi-GPU Wan2.2-I2V)
54+
55+ ``` fish
56+ #!/usr/bin/env fish
57+ # Run from the lmms-eval repo root.
58+ cd /path/to/lmms-eval; or exit 1
59+
60+ # Rule-based VBVR scorers read the GT mp4s/pngs from this root.
61+ set -gx VBVR_GT_PATH /path/to/VBVR-Bench
62+
63+ set MODEL_DIR /path/to/Wan2.2-I2V-A14B-Diffusers
64+ set OUT_ROOT /path/to/eval_out/vbvr_wan22_full_highres
65+ set VIDEOS_DIR $OUT_ROOT/videos
66+ set METRICS_DIR $OUT_ROOT/metrics
67+ mkdir -p $VIDEOS_DIR $METRICS_DIR
68+
69+ set MODEL_ARGS "model=$MODEL_DIR"
70+ set MODEL_ARGS "$MODEL_ARGS,output_dir=$VIDEOS_DIR"
71+ set MODEL_ARGS "$MODEL_ARGS,data_parallel=4,num_gpus=2,sp_size=2,tp_size=1"
72+ set MODEL_ARGS "$MODEL_ARGS,num_inference_steps=50,num_frames=81"
73+ set MODEL_ARGS "$MODEL_ARGS,height=1024,width=1024,fps=16"
74+ set MODEL_ARGS "$MODEL_ARGS,dit_cpu_offload=False,text_encoder_cpu_offload=True"
75+ set MODEL_ARGS "$MODEL_ARGS,image_encoder_cpu_offload=False,vae_cpu_offload=False"
76+ set MODEL_ARGS "$MODEL_ARGS,enable_torch_compile=True"
77+
78+ exec stdbuf -oL -eL .venv/bin/python -m lmms_eval eval \
79+ --model fastvideo \
80+ --model_args $MODEL_ARGS \
81+ --tasks vbvr \
82+ --batch_size 1 \
83+ --log_samples \
84+ --output_path $METRICS_DIR
85+ ```
86+
87+ Generated videos land in ` $VIDEOS_DIR ` ; per-sample logs and aggregated metrics
88+ land in ` $METRICS_DIR ` . Tune ` data_parallel ` , ` num_gpus ` , ` sp_size ` , and the
89+ ` *_cpu_offload ` flags to match your hardware.
90+
5391## Metrics
5492
5593- ` vbvr_overall ` — sample-weighted mean across both splits
0 commit comments