Fix macro/weighted F1 and two sibling averaging bugs in deep.metric - #877
Merged
Merged
Conversation
Follow-up to haifengl#874, which fixed the same F-of-means defect in smile.validation.metric.FScore. Auditing the whole deep.metric package against scikit-learn turned up three variants of one mistake: the reduction of an averaging strategy is applied to the wrong quantity, or without the guard the other strategies have. F1Score.compute() built the macro and weighted score as 2PR/(P+R) from the already averaged precision and recall. F1 is not linear in (P, R), so that is not an F1 score of anything. On a 3-class set with per-class F1 = [4/7, 2/5, 1] it returned 0.774 for a macro F1 of 23/35 = 0.657. Compute the per-class F1 first, then average. Recall.compute() divided the micro score by the per-class sizes after the zero-size guard had replaced empty classes by one, so a class with no instance in the batch inflated the denominator and micro recall no longer equalled accuracy, contrary to the Averaging javadoc. Micro divides by the sample total. The weighted reduction had no guard for an empty state, so Precision, Recall and F1Score all returned NaN after reset(), where macro and micro return 0. Averaging.weighted() now performs that reduction once for the three metrics. Precision.score() and Recall.score() expose the per-class score so that F1Score can average it without duplicating the guards. Verified against scikit-learn 1.8.0 over 10 label sets (imbalanced 3 and 4 class, binary, never-predicted class, empty class, multi-batch, random 5-class): 18 divergences before, none after, largest remaining difference 9e-08. The existing tests missed all of this because they only cover perfect predictions, binary classification and micro averaging on batches where every class is present.
Owner
|
Thanks a lot! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #874, as requested there.
deep.metric.F1Scorehad the same F-of-means defect that #874 fixed insmile.validation.metric.FScore. While auditing the rest of the package against scikit-learn I found two more variants of the same mistake — the reduction of an averaging strategy applied to the wrong quantity, or without the guard the other strategies have — so this fixes the class rather than the one method.1. Macro and weighted F1 averaged the wrong thing
F1Score.compute()calledprecision.compute()andrecall.compute(), which already reduce the per-class vector to a scalar, then took their harmonic mean. F1 is not linear in (P, R), soF1(mean P, mean R)is not the mean of the per-class F1 scores.Confusion matrix (rows = truth, columns = prediction):
per-class precision = [1, 1/4, 1], recall = [2/5, 1, 1], F1 = [4/7, 2/5, 1]
2. Micro recall stopped equalling accuracy when a class is empty
Recall.compute()divided the micro score bywhere(size > 0, size, 1).sum(). That guard exists to keep the per-class recall of an empty class at 0 instead of NaN, but summing it makes every empty class add one phantom sample to the micro denominator.Averaging.Micro's javadoc states that micro precision and recall equal accuracy, andvalidation.metric.Recalldivides byn.With 3 output classes, no instance of class 2 in the batch and 8 of 10 samples correct, micro recall was 8/11 = 0.72727275 and micro F1 was 16/21 = 0.76190478, against an accuracy of 0.8. Any mini-batch that happens to miss a rare class hits this.
3. Weighted metrics returned NaN after reset()
The weighted reduction
score.mul(size).sum().div(size.sum())has no guard for the empty state, soPrecision,RecallandF1Scoreall returned NaN afterreset()— called at the end of every epoch, per theMetricjavadoc. Macro and micro return 0 there, and there is already a test asserting that for macro F1.Changes
Precision.score()andRecall.score()expose the per-class score with the guard each already had, soF1Scorecan average the per-class F1 without duplicating them.Averaging.weighted()performs the support-weighted reduction once for the three metrics.Recall.compute()now branches on the strategy rather than ontp.size(0).Verification
Differential run against scikit-learn 1.8.0 over 10 label sets (perfect, imbalanced 3-class 900/80/20, imbalanced 4-class, binary, never-predicted class, empty class, all-wrong, multi-batch, random 5-class, single-class truth), covering every metric and averaging strategy — 109 comparisons. 18 divergences before, none after, largest remaining difference 9e-08 in float32. Accuracy, all three precisions, macro/weighted recall and the binary metrics already agreed, so the audit closes the class.
7 new tests in
MetricTest, every one of them failing before the change../gradlew :deep:testgoes from 391 tests / 0 failures to 398 / 0. The existing tests missed all three bugs because they only cover perfect predictions, binary classification, and micro averaging on batches where every class is present.