Add BrierScore and ECE classification calibration metrics#1896
Open
bhargavi1045 wants to merge 1 commit into
Open
Add BrierScore and ECE classification calibration metrics#1896bhargavi1045 wants to merge 1 commit into
bhargavi1045 wants to merge 1 commit into
Conversation
- New ClassificationCalibrationMetrics legacy metric computes Brier score and ECE from prediction probabilities, as a standalone metric rather than extending the shared ClassificationQualityMetric - Wrap as two v2 metrics BrierScore and ECE (ECE exposes n_bins param) following the existing RocAuc/LogLoss bridging pattern - Register type aliases for both legacy and v2 serialization - Add parametrized test coverage and multitest legacy fixture
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1895
What
Add two new classification metrics for evaluating probability calibration:
BrierScore: mean squared error between predicted probabilities and actual outcomes. Lower is better. Ranges from 0 (perfect) to 1 (worst). Works for binary and multiclass.ECE(Expected Calibration Error) : bins predictions by confidence and compares each bin's average confidence to its actual accuracy. Lower is better. Exposes ann_binsparameter (default 10).Why
Evidently has good coverage of discrimination metrics (ROC AUC, F1, precision, recall) but no calibration metrics. Brier Score and ECE are widely used calibration metrics for evaluating whether predicted probabilities are well calibrated, rather than only whether the model ranks predictions correctly.
Usage
Implementation notes
ClassificationCalibrationMetricscomputes both Brier score and ECE in a single pass and avoids touching the sharedClassificationQualityMetricBrierScoreandECEfollow the existingRocAuc/LogLossbridging patternlegacy/metrics/_registry.pyandcore/registries/metrics.pytests/future/metrics/all_metrics_tests.py)tests/multitest/metrics/classification.py)Files changed
src/evidently/legacy/metrics/classification_performance/calibration_metric.py(new)src/evidently/legacy/metrics/__init__.pysrc/evidently/legacy/metrics/_registry.pysrc/evidently/metrics/classification.pysrc/evidently/metrics/__init__.pysrc/evidently/core/registries/metrics.pytests/future/metrics/all_metrics_tests.pytests/multitest/metrics/classification.py