feat: Support FastVideo for Video Generation Models

[pufanyi]

Added FastVideo and VBVR

#!/usr/bin/env fish
# Run from the lmms-eval repo root.
cd /mnt/umm/users/pufanyi/workspace/lmms-eval; or exit 1

# Rule-based VBVR scorers read the GT mp4s/pngs from this root.
set -gx VBVR_GT_PATH /mnt/umm/users/pufanyi/workspace/Wan-Trainer/storage/datasets/VBVR-Bench

set MODEL_DIR   /mnt/umm/users/pufanyi/workspace/Wan-Trainer/storage/models/Wan2.2-I2V-A14B-Diffusers
set OUT_ROOT    /mnt/umm/users/pufanyi/workspace/Wan-Trainer/storage/eval_out/vbvr_wan22_full_highres
set VIDEOS_DIR  $OUT_ROOT/videos
set METRICS_DIR $OUT_ROOT/metrics
mkdir -p $VIDEOS_DIR $METRICS_DIR

set MODEL_ARGS "model=$MODEL_DIR"
set MODEL_ARGS "$MODEL_ARGS,output_dir=$VIDEOS_DIR"
set MODEL_ARGS "$MODEL_ARGS,data_parallel=4,num_gpus=2,sp_size=2,tp_size=1"
set MODEL_ARGS "$MODEL_ARGS,num_inference_steps=50,num_frames=81"
set MODEL_ARGS "$MODEL_ARGS,height=1024,width=1024,fps=16"
set MODEL_ARGS "$MODEL_ARGS,dit_cpu_offload=False,text_encoder_cpu_offload=True"
set MODEL_ARGS "$MODEL_ARGS,image_encoder_cpu_offload=False,vae_cpu_offload=False"
set MODEL_ARGS "$MODEL_ARGS,enable_torch_compile=True"

exec stdbuf -oL -eL .venv/bin/python -m lmms_eval eval \
    --model fastvideo \
    --model_args $MODEL_ARGS \
    --tasks vbvr \
    --batch_size 1 \
    --log_samples \
    --output_path $METRICS_DIR

There is a minor issue: the current final process results in VBVR are single-threaded. If we want to change this to multi-threaded, it seems we would need to modify the main trunk of the code. The final process results step takes approximately 10 minutes to run. However, if you use 32 threads, it can be completed within a minute.

lmms-eval/lmms_eval/tasks/vbvr/utils.py

Lines 191 to 192 in 545ffdf

	evaluator = get_evaluator(task_name)
	vr = evaluator.evaluate(eval_info, task_specific_only=True)

[kcz358]

This part should utilize the cache directory and the submission directory. Using environment variables seems rather inflexible. After examining the repo structure on HuggingFace, one possibility is to set up a cache directory similar to the other video paths. Then, before each download, we can use snapshot_download from HuggingFace, which would return the path to the video directory. As for the metrics directory, I think it could be handled via generate_submission?

[pufanyi]

Oh just find forgot to write the downloading code, updated by adding

lmms-eval/lmms_eval/tasks/vbvr/utils.py

Line 86 in cc5267e

snapshot_root = snapshot_download(repo_id=_dataset_repo_id(), repo_type="dataset")

[pufanyi]

Lemme check the model output. I got lazy before and just pointed it to my training folder randomly. Let me fix that now.

[pufanyi]

Seems that 笨蛋 claude makes the default output dir to huggingface dir

lmms-eval/lmms_eval/models/chat/fastvideo.py

Lines 73 to 75 in cc5267e

	def _default_output_dir() -> str:
	hf_home = os.path.expanduser(os.getenv("HF_HOME", "~/.cache/huggingface"))
	return os.path.join(hf_home, "lmms_eval", "generated_videos", "fastvideo")

This definitely needs to be changed.

But I’m not entirely sure how to handle generate_submission. I noticed that Bagel outputs to ./logs/bagel_images/<run_id>/ by default, so maybe we should do the same?

[pufanyi]

Fixed by adding

lmms-eval/lmms_eval/models/chat/fastvideo.py

Lines 81 to 87 in c1019e4

	def _model_slug(model_path: str) -> str:
	base = os.path.basename(str(model_path).rstrip("/"))
	return _SAFE_RE.sub("_", base).strip("_") or "model"
	def _default_output_dir(model_path: str) -> str:
	return os.path.join("./logs/fastvideo", _model_slug(model_path), _generate_run_id())

[kcz358]

Yeah, I think currently could use log dir as the output dir, unless otherwise specify in the init args.

[kcz358]

Maybe can the caching features for lmms-eval instead of hardcoding here? I am fine for this, just wondering if this is possible.

[pufanyi]

What I’m actually more concerned about is that right now the videos and the video paths are separated. Videos don’t seem as easy to cache as plain text. But I guess it’s still doable — worst case, it just throws a “video not found” error.

[kcz358]

I think if we store the video as path in the output, the caching logic would just like plain text? We have a similar structure just like text so can just load from db. Or if this is not the case can just reload from the previously define output dir and keep this code block

[pufanyi]

ok, then I will do this. I was actually concerned about the scenario where the path still exists but the video has been deleted. My understanding is that our script doesn't check for this.

[kcz358]

Yeah, I think our script doesn't check for this. If the storage is not persistent then maybe should disable the cache mode or clean the cached data.

[pufanyi]

I made the changes. I'll run it tonight to check the if the results are the same.

[kcz358]

Ok thanks, I think once you feel this PR is mostly done, I will approve and merge the PR. Thanks!

[pufanyi]

Seems that the result is ok.