Metrics for uncertainty comparison

[deebuls]

RER = Remaining Error rate
RAR = Remaining Accuracy rate

	Certain	Uncertain
Correct	CC	UC
Incorrect	CI	UI

A system which relies on these estimates is expected to be
in functional mode if the predictions are certain and in 
fallback/mitigation mode if the prediction is uncertain. 
However, the most critical result regarding safety are the 
predictions where the model is certain about its prediction 
but incorrect (CI). We call the ratio of the number of certain but
incorrect samples to all samples the Remaining Error Rate
(RER). For minimizing the overall risk, it needs to be as low
as possible. Nonetheless, if a model would always give a low
confidence as output, the system would constantly remain in
fall-back mode and will be unable to provide the intended
functionality. Therefore, the ratio of the number of certain
and correct samples to all samples - we call it the Remaining Accuracy Rate (RAR) - needs to be as high as possible
to stay in performance mode for most of the time.

RER = CI / (CC+UC+CI+UI) ⬇️
RAR = CC / (CC+UC+CI+UI) ⬆️

https://ceur-ws.org/Vol-2560/paper35.pdf

Benchmarking Uncertainty Estimation Methods for Deep Learning With
Safety-Related Metrics
Maximilian Henne, Adrian Schwaiger, Karsten Roscher, Gereon Weiss