Issue reproducing DIVA for OpenAI CLIP (ViT-L/14-224) on CC3M: Performance does not surpass the original baseline

[wzd0328]

Dear Authors,

Thank you for your excellent work and for open-sourcing the code!

We are currently trying to reproduce the results of DIVA using OpenAI CLIP (ViT-L/14-224) on the CC3M dataset. However, we have encountered a problem: after the training process is complete, the performance of our trained model does not surpass that of the original OpenAI CLIP.

To help identify what might be going wrong, we have recorded our training and evaluation logs here:

1. config.json:
   {
   "actual_bs": 1,
   "architectures": [
   "SDModel"
   ],
   "clip_image_size": 224,
   "input": {
   "batch_size": 1,
   "mean": [
   0.485,
   0.456,
   0.406
   ],
   "model_type": "",
   "std": [
   0.229,
   0.224,
   0.225
   ]
   },
   "metaclip_version": 1,
   "model": {
   "adapt_only_classifier": true,
   "freeze_class_embeds": true,
   "freeze_vae": false,
   "model_type": "",
   "override_total_steps": -1,
   "sd_version": "2-1",
   "use_flash": false
   },
   "output_dir": "./outputs/First_Start",
   "torch_dtype": "float32",
   "transformers_version": "4.39.3",
   "tta": {
   "adapt_topk": -1,
   "gradient_descent": {
   "accum_iter": 1,
   "optimizer": "sgd",
   "optimizer_momentum": 0.9,
   "train_steps": 2,
   "weight_decay": 0
   },
   "loss": "mse",
   "model_type": "",
   "rand_timestep_equal_int": false,
   "random_timestep_per_iteration": true,
   "use_same_noise_among_timesteps": false
   },
   "visual_pattern": "None"
   }

2.pred_result.json:
{
"results_before": {
"openai-ViT-L-14": {
"Orientation and Direction": 0.0,
"Presence of Specific Features": 6.666666666666667,
"State and Condition": 26.666666666666668,
"Quantity and Count": 13.333333333333334,
"Positional and Relational Context": 6.666666666666667,
"Color and Appearance": 40.0,
"Structural Characteristics": 33.33333333333333,
"Texts": 13.333333333333334,
"Viewpoint and Perspective": 20.0,
"average_score": 17.77777777777778
}
},
"results_final_after": {
"openai-ViT-L-14": {
"Orientation and Direction": 6.666666666666667,
"Presence of Specific Features": 13.333333333333334,
"State and Condition": 13.333333333333334,
"Quantity and Count": 6.666666666666667,
"Positional and Relational Context": 13.333333333333334,
"Color and Appearance": 40.0,
"Structural Characteristics": 26.666666666666668,
"Texts": 6.666666666666667,
"Viewpoint and Perspective": 20.0,
"average_score": 16.296296296296298
}
}
}

Furthermore, we checked the trainer_state.json and noticed that the training loss did not converge.

Any insights or suggestions would be greatly appreciated. Thank you very much for your time and help!