Skip to content

Add BrierScore and ECE classification calibration metrics#1896

Open
bhargavi1045 wants to merge 1 commit into
evidentlyai:mainfrom
bhargavi1045:add-brier-score-ece-metrics
Open

Add BrierScore and ECE classification calibration metrics#1896
bhargavi1045 wants to merge 1 commit into
evidentlyai:mainfrom
bhargavi1045:add-brier-score-ece-metrics

Conversation

@bhargavi1045

Copy link
Copy Markdown

Closes #1895

What

Add two new classification metrics for evaluating probability calibration:

  • BrierScore : mean squared error between predicted probabilities and actual outcomes. Lower is better. Ranges from 0 (perfect) to 1 (worst). Works for binary and multiclass.
  • ECE (Expected Calibration Error) : bins predictions by confidence and compares each bin's average confidence to its actual accuracy. Lower is better. Exposes an n_bins parameter (default 10).

Why

Evidently has good coverage of discrimination metrics (ROC AUC, F1, precision, recall) but no calibration metrics. Brier Score and ECE are widely used calibration metrics for evaluating whether predicted probabilities are well calibrated, rather than only whether the model ranks predictions correctly.

Usage

from evidently.metrics import BrierScore, ECE
from evidently import Report

report = Report(metrics=[BrierScore(), ECE(n_bins=10)])
report.run(current_data=dataset, reference_data=None)

Implementation notes

  • New standalone legacy metric ClassificationCalibrationMetrics computes both Brier score and ECE in a single pass and avoids touching the shared ClassificationQualityMetric
  • Two thin v2 wrappers BrierScore and ECE follow the existing RocAuc/LogLoss bridging pattern
  • Type aliases registered in both legacy/metrics/_registry.py and core/registries/metrics.py
  • Full parametrized test coverage added (tests/future/metrics/all_metrics_tests.py)
  • Legacy multitest fixture added (tests/multitest/metrics/classification.py)

Files changed

  • src/evidently/legacy/metrics/classification_performance/calibration_metric.py (new)
  • src/evidently/legacy/metrics/__init__.py
  • src/evidently/legacy/metrics/_registry.py
  • src/evidently/metrics/classification.py
  • src/evidently/metrics/__init__.py
  • src/evidently/core/registries/metrics.py
  • tests/future/metrics/all_metrics_tests.py
  • tests/multitest/metrics/classification.py

- New ClassificationCalibrationMetrics legacy metric computes Brier
  score and ECE from prediction probabilities, as a standalone metric
  rather than extending the shared ClassificationQualityMetric
- Wrap as two v2 metrics BrierScore and ECE (ECE exposes n_bins param)
  following the existing RocAuc/LogLoss bridging pattern
- Register type aliases for both legacy and v2 serialization
- Add parametrized test coverage and multitest legacy fixture
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add BrierScore and ECE as classification calibration metrics

1 participant