Question about plotting calibration curve for custom layers

[sandeeppandey456]

Hello sir,

I have a model comprising of custom layers ( something inlines of CNN) . I have saved the weights of the model in h5 file. It would be very helpful if yu can demonstrate/ suggest how to use the calibration code for models with custom layer.

[markus93]

Hey,

sorry for a late answer.

In the script https://github.com/markus93/NN_calibration/blob/master/scripts/utility/evaluation.py is a method "evaluate_model". You can use this to get the logits out from the model and it also saves the logits into the file.

As an input you need:

1. the model, it may have custom layers. (the last layer should be softmax layer)
2. Weights' file, so the path to the weights (.h5 is good)
3. Datasets for testing and validation. The validation set is used for training calibration algorithm. And the test set is for evaluating the calibration algorithm.

You can also write Your custom code based on this to get logits. However, logits are only needed for scaling methods (i.e temperature scaling, vector scaling, matrix scaling). If You want to use some other methods, it is easier just to let the model predict based on the input and then use the output of the softmax layer.

After you have saved pickled logits, you can check the https://github.com/markus93/NN_calibration/blob/master/scripts/calibration/Calibration%20-%20(Temp%2C%20Iso%2C%20Beta%2C%20Hist).ipynb to get an overall idea what is going on.

Let me know if You have any other questions :)