Reproduce VGT-AR Pretraining (InternVL3)

[DingShizhe]

I’m pretraining VGT InternVL3 0.6B (448px) using the official pretraining pipeline, but the generation quality is much lower than expected. Training appears stable and the loss decreases normally, but visual quality remains poor and improves only slightly over time.

I attach generated samples from 2k, 50k, and 100k iterations, as well as the loss curves.

samples from 2k iterations

samples from 50k iterations

samples from 100k iterations

loss curves

---

Training Setup

• 100k iterations, 8 GPUs (DDP)
• Global batch size: 256
• LR: 3e-4 peak, cosine decay to 1e-4
• Warmup: 1k iters
• AdamW (0.9, 0.95), weight decay 0.05
• EMA start 10k (momentum 0.0002)
• REPA loss weight 0.5

Config: configs/pretrain/vgt_internvl3_0_6B_448px_pretrain.py

---

Dataset

Mixed training data:

• megalith10m
• text2image2m
• imagenet1k_t2i_qwenvl_flux

---

Question

Is this level of generation quality expected for this setup, or is there anything important I should check or adjust?

Thanks — happy to share more details if needed.

[gjhhust]

Thank you for your attention. The training seems to be normal, but the pre-training data of about 12M appears to be too little.，You need to add more data like datasets, The low-quality data from megalith10m accounts for an excessively large proportion in your current ratio.

[DingShizhe]

So if I understand correctly, I can combine these datasets—cc12m, text2image2m, redcaps5m, laion6m, imagenet1k_t2i_qwenvl_flux—for pretraining and get promising results, right?

Is midjourney_captioned_23m really necessary? I’d like to use it, but it’s just too big.

[gjhhust]

If your space is limited, you can try laion6m, text2image2m, midjourney (download part of it), and more you can try cc12m as pretrain, and then fine-tune.

[DingShizhe]

Thanks so much for the suggestion!

Just a quick follow-up — could you share more details about the datasets you used for pretraining the model? Are they the ones you mentioned here: laion6m, text2image2m, midjourney, and cc12m, so I can reliably reproduce your setup?

And for the training setup — was it 50k iterations with a batch size of 512?

If possible, could I also see how vgt-internvl performs after pretraining only (without finetuning), in terms of generated images?

That would be extremely helpful — I’d really appreciate it!

[gjhhust]

Sure! Please contact gjh_hust@foxmail.com