Skip to content

Fix macro/weighted F1 and two sibling averaging bugs in deep.metric - #877

Merged
haifengl merged 1 commit into
haifengl:masterfrom
gaoflow:fix-deep-metric-averaging
Jul 26, 2026
Merged

Fix macro/weighted F1 and two sibling averaging bugs in deep.metric#877
haifengl merged 1 commit into
haifengl:masterfrom
gaoflow:fix-deep-metric-averaging

Conversation

@gaoflow

@gaoflow gaoflow commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #874, as requested there.

deep.metric.F1Score had the same F-of-means defect that #874 fixed in smile.validation.metric.FScore. While auditing the rest of the package against scikit-learn I found two more variants of the same mistake — the reduction of an averaging strategy applied to the wrong quantity, or without the guard the other strategies have — so this fixes the class rather than the one method.

1. Macro and weighted F1 averaged the wrong thing

F1Score.compute() called precision.compute() and recall.compute(), which already reduce the per-class vector to a scalar, then took their harmonic mean. F1 is not linear in (P, R), so F1(mean P, mean R) is not the mean of the per-class F1 scores.

Confusion matrix (rows = truth, columns = prediction):

        pred 0  pred 1  pred 2   support
true 0       4       6       0        10
true 1       0       2       0         2
true 2       0       0       3         3

per-class precision = [1, 1/4, 1], recall = [2/5, 1, 1], F1 = [4/7, 2/5, 1]

before correct
macro F1 0.77419355 23/35 = 0.65714286
weighted F1 0.72 111/175 = 0.63428571

2. Micro recall stopped equalling accuracy when a class is empty

Recall.compute() divided the micro score by where(size > 0, size, 1).sum(). That guard exists to keep the per-class recall of an empty class at 0 instead of NaN, but summing it makes every empty class add one phantom sample to the micro denominator. Averaging.Micro's javadoc states that micro precision and recall equal accuracy, and validation.metric.Recall divides by n.

With 3 output classes, no instance of class 2 in the batch and 8 of 10 samples correct, micro recall was 8/11 = 0.72727275 and micro F1 was 16/21 = 0.76190478, against an accuracy of 0.8. Any mini-batch that happens to miss a rare class hits this.

3. Weighted metrics returned NaN after reset()

The weighted reduction score.mul(size).sum().div(size.sum()) has no guard for the empty state, so Precision, Recall and F1Score all returned NaN after reset() — called at the end of every epoch, per the Metric javadoc. Macro and micro return 0 there, and there is already a test asserting that for macro F1.

Changes

Precision.score() and Recall.score() expose the per-class score with the guard each already had, so F1Score can average the per-class F1 without duplicating them. Averaging.weighted() performs the support-weighted reduction once for the three metrics. Recall.compute() now branches on the strategy rather than on tp.size(0).

Verification

Differential run against scikit-learn 1.8.0 over 10 label sets (perfect, imbalanced 3-class 900/80/20, imbalanced 4-class, binary, never-predicted class, empty class, all-wrong, multi-batch, random 5-class, single-class truth), covering every metric and averaging strategy — 109 comparisons. 18 divergences before, none after, largest remaining difference 9e-08 in float32. Accuracy, all three precisions, macro/weighted recall and the binary metrics already agreed, so the audit closes the class.

7 new tests in MetricTest, every one of them failing before the change. ./gradlew :deep:test goes from 391 tests / 0 failures to 398 / 0. The existing tests missed all three bugs because they only cover perfect predictions, binary classification, and micro averaging on batches where every class is present.

Follow-up to haifengl#874, which fixed the same F-of-means defect in
smile.validation.metric.FScore. Auditing the whole deep.metric package
against scikit-learn turned up three variants of one mistake: the
reduction of an averaging strategy is applied to the wrong quantity, or
without the guard the other strategies have.

F1Score.compute() built the macro and weighted score as
2PR/(P+R) from the already averaged precision and recall. F1 is not
linear in (P, R), so that is not an F1 score of anything. On a 3-class
set with per-class F1 = [4/7, 2/5, 1] it returned 0.774 for a macro F1
of 23/35 = 0.657. Compute the per-class F1 first, then average.

Recall.compute() divided the micro score by the per-class sizes after
the zero-size guard had replaced empty classes by one, so a class with
no instance in the batch inflated the denominator and micro recall no
longer equalled accuracy, contrary to the Averaging javadoc. Micro
divides by the sample total.

The weighted reduction had no guard for an empty state, so
Precision, Recall and F1Score all returned NaN after reset(), where
macro and micro return 0. Averaging.weighted() now performs that
reduction once for the three metrics.

Precision.score() and Recall.score() expose the per-class score so that
F1Score can average it without duplicating the guards.

Verified against scikit-learn 1.8.0 over 10 label sets (imbalanced 3
and 4 class, binary, never-predicted class, empty class, multi-batch,
random 5-class): 18 divergences before, none after, largest remaining
difference 9e-08. The existing tests missed all of this because they
only cover perfect predictions, binary classification and micro
averaging on batches where every class is present.
@haifengl

Copy link
Copy Markdown
Owner

Thanks a lot!

@haifengl
haifengl merged commit 8250509 into haifengl:master Jul 26, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants