Recreate Table Result for Trained MobileOne S0 Model

[aidansmyth95]

Hi there,

I trained MobileOneS0 from scratch for 200 epochs, on the Gaze360 dataset.

The result I get from evaluating that model on the Gaze360 dataset is MAE of 18.619, which is quite a bit different from the reported results in the README table.

Did I miss some other params that need setting? I left everything else at default in the training script (num-epochs 200, modified my batch-size to be 64 due to memory constraints, but that shouldn't matter).

I really appreciate any help you can provide. Can anyone recreate the table results for any model, or do they all need updating?

[yakhyo]

Hi @aidansmyth95,

Sorry for late reply. I am not sure why you got that low score. Have you evaluated the best model as well. It saves two type of weights based on the loss: best and last weights. Try the best weight as well.

[aidansmyth95]

Hi @yakhyo ,

So when I evaluate the pretrained models, I get the expected results. So this verifies that the data curation and preprocessing is correct, and that we are using the right data when training the models.

Now my question would be, can you share more detailed info on how you trained those models? Because I think something in this repo has changed since then. For MobileOne_S0, I notice poor generalization, and for MobileNetv2, I notice it quickly overfits.

Did you train the models using a different repo? If not, which commit of this repo is closest to when those models were trained? Was alpha parameter and learning rate the same as the argparse defaults we see in today's main.py script?

Thanks in advance!

[yakhyo]

Hi @aidansmyth95

I have trained the models using this repo only. Yes MobileOne_S0 has poor results and as far as I remember, it was not stable at all. Maybe due to the reparameterization and the model itself was small.

For the case of MobileNet V2, did you check the best.pt instead of the last.pt. The models I uploaded are the best models only.

I did not change any parameters and I trained everything as it is.

When you train the models did you evaluate them and reproduced the results on the readme ?