whether we need to pay attention to some details during training?

[AdamZHAOZhou]

Hi,
　I downloaded your code and run it, but I can't get your results. Do I need to pay attention to certain details during the training process? And is your model easy to train?
Looking forward to your reply.

[SharifAmit]

Hi, could you elaborate the problem you are facing?

As far as the training goes, we don't change any hyperparameters during training.

[AdamZHAOZhou]

until now, I have trained 80 epoch, and I did not change any hyperparameters, but the results of F1 score was still less than 0.8.

[AdamZHAOZhou]

I downloaded your weight and test, the results was right, so I did not whether I need to pay attention to some details during training.

[SharifAmit]

Hi, you are on the right track. After training, loop over all the models and then print the metics for each to find the best snapshot. As GAN doesn't converge like regular model, the loss values fluctuate a lot, so the last saved model won't be the best one.

You can also try to reload the best snapshot and train again. I believe I did it for some of my similar work, as the discriminator might suffer from mode collapse and only print loss=0.00 . This is a common phenomenon with GAN training.

Also, while testing, try to use a smaller stride ( stride=3) to do the patch-wise prediction on the whole test image. This is slow but will you give you the best auc and F1 score.

Hope this helps!

[AdamZHAOZhou]

I tested each epoch, but the F1 score was still less than 0.80.
I also tried a smaller stride (stride=3), but not much improvement．
Could you tell me about your training process?

[SharifAmit]

I have trained the model for 100 epochs. After training, I loaded the weights of the best performing Coarse and Fine Generator ( For the same epoch) and then trained again for 100 epochs. I did not load the weights of the discriminators while retraining. So the discriminators were trained from scratch. I think I did this procedure 2-3 times. And also looped over all 100 saved weights to find the best performing coarse and fine generator pairs.

Thanks !

[AdamZHAOZhou]

For table 1 in your paper, I used sensitivity,specificity and accuracy to calculate the F1 score on CHASE-DB1, and I got 0.7858 F1 score, but your result is 0.8957. M-GAN's sensitivity,specificity and accuracy better than your RV-GAN, however, F1 score is 0.08 lower than yours. Could you explain this?

[SharifAmit]

ChaseDB doesnt come with training and test splits. Which images are you using for Chase-DB for testing ? Maybe the evaluation technique + test images you are using is giving you the bad results?

Also the table results are correct, as we have experimented and validated the weights extensively with different stride values, and found stride=3 to give us the best results. The only typo is the M-GAN sensitivity result will be in BOLD text rather than ours.

[AdamZHAOZhou]

I used sensitivity=0.8199, specificity=0.9806 and accuracy= 0.9697 to calculate F1 score, as shown in table 1, however, the value of F1 score I calculated by hand is too far from the value of F1 score in the table 1.

[SharifAmit]

F1-scrore is calculated using Precision and Sensitivity. We used pycm library.

import pycm

cm = ConfusionMatrix(actual_vector=y_true, predict_vector=y_pred)

print("Specificity: " + str(cm.TNR[1]))
print("Sensitivity: " + str(cm.TPR[1]))
print("Accuracy : " + str(cm.ACC[0]))
print("F1 Score: " + str((2*cm.PPV[0]*cm.TPR[1])/(cm.PPV[0]+cm.TPR[1])))

[AdamZHAOZhou]

I think you should check the calculation again instead of the program. Because I also tested M-GAN by sensitivity=0.8234, specificity=0.9938 and accuracy= 0.9736 to calculate F1 score, which is right.

[smridhi29]

I downloaded your weight and test, the results was right, so I did not whether I need to pay attention to some details during training.

Where did you test the weights and biases? Could you please help me with the procedure?