No Checkpoint found after running run_128_B.sh

[wd255]

For the reason of #17 I turned off the sanity check in the Trainer code to avoid doing validation.

The training finished as I got some tfevent log files that look correct, but

1. Validation failed as expected because of Training phase sanity check fails by loading "../../data/imagenet/val" as an image #17
2. I find nothing in my checkpoint path(It's an existing directory on my machine, set here: https://github.com/TencentARC/Open-MAGVIT2/blob/main/configs/imagenet_lfqgan_128_B.yaml#L15). The total step is around 16000.

The potential causes I can think of are

1. validation failed and checkpoint saving is part of validation or depends on it
2. Total step not enough for a checkpoint
3. I'm not configuring the checkpoint save path correctly

Has anyone encountered this error?

[SekeunKim]

same problem here, i cannot find ckpt path after training.

[SekeunKim]

There is missing log file config in yaml file.
Add this
callbacks:
- class_path: lightning.pytorch.callbacks.ModelCheckpoint
init_args:
dirpath: "./checkpoints/vqgan/test"
filename: "best-{epoch}-{val_loss:.4f}"
monitor: "val_ema/reconstruct_loss"
mode: "min"
save_top_k: 1
save_last: true

Based on what you monitor in .yaml file.