[question] how to get teacher model for Turbo-VAE-DC?

[eisneim]

i'm trying to copy DCAE class to turbo-vaed from https://github.com/hpcaitech/Open-Sora/tree/main ckpt using https://huggingface.co/hpcai-tech/Open-Sora-v2-Video-DC-AE/tree/main

device = "cuda" if torch.cuda.is_available() else "cpu"
frame_count = 17 # Tipical for video vae: factor * N + 1
dtype = torch.float16

model_config = "configs/Turbo-VAED-DC.json"
resume_from_checkpoint = "pretrained/Turbo-VAED-DC.pth"
teacher_model_name = "DCAE"
# downloaded from huggingface:  https://huggingface.co/hpcai-tech/Open-Sora-v2-Video-DC-AE/tree/main
dcae_state_dict = "path_to_/Turbo-VAED/pretrained/F32T4C128_AE.pt"
from dc_ae.models.dc_ae import dc_ae_f32, DCAE
dcae_config = dc_ae_f32("dc-ae-f32t4c128", dcae_state_dict)
teacher_model = DCAE(dcae_config).to(device, dtype=dtype)


model = AutoencoderKLTurboVAED.from_config(
    config=load_json_to_dict(model_config)
)
checkpoint = torch.load(resume_from_checkpoint, map_location="cpu")
model.decoder.load_state_dict(checkpoint, strict=False)
model = model.to(device, dtype=dtype)

when i run i got

时间维度必须是下采样因子的倍数....

i was wonder if i'm using the correct DC-AE model

[eisneim]

if i set frame number to even number like 16:

output reconstruction video is in grid pattern, any instruction on how to make DC version of turbo-vaed work? i used LTX vae before, the 8x reduction on frame dimension is too agressive, 4x is the sweet spot

[ZouYa99]

Hi,
For Video DC-AE, due to the original model's configuration, the number of frames in the input video must be a multiple of 4.
Additionally, the address where you downloaded the code and weights for the Video DC-AE model is correct. The relevant loading code can be found at #5 (comment). If issues persist, try using float32.
If you have any further questions, please feel free to ask. :)