feat(report): add calibration diagnostics

AlisinaDevelo · AlisinaDevelo · commit d389700eac81 · 2026-07-01T04:30:06.000+02:00
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -13,6 +13,7 @@
 - `quorabust-train --metadata-out` for JSON metadata sidecars.
 - Thresholded `/predict` decisions with artifact/env defaults and per-request override.
 - Enterprise product-positioning guide for semantic matching use cases and production gaps.
+- Calibration summary and probability-bin diagnostics in `quorabust-report`.
 
 ### Fixed
 - Capped NumPy below 2.5 so Mypy can parse dependency stubs with the configured
diff --git a/README.md b/README.md
@@ -78,7 +78,8 @@ quorabust-report \
 ```
 
 The report includes artifact metadata, persisted training/eval metrics, optional holdout
-metrics, a confusion matrix, and a precision/recall/F1 threshold sweep. Use
+metrics, a confusion matrix, a precision/recall/F1 threshold sweep, and probability
+calibration diagnostics. Use `--calibration-bins` to tune the calibration table and
 `--format json` for machine-readable CI or release artifacts. Use a real held-out CSV for
 comparable model claims; the command accepts the same `question1`, `question2`,
 `is_duplicate` column contract as training.
diff --git a/docs/REPORTING.md b/docs/REPORTING.md
@@ -2,7 +2,8 @@
 
 Use `quorabust-report` to turn a saved artifact into a Markdown model card. The report
 is meant for artifact review: it shows lineage, feature schema, persisted metrics, and
-optional holdout metrics with a threshold confusion matrix.
+optional holdout metrics with a threshold confusion matrix, threshold sweep, and
+probability calibration summary.
 
 ## Smoke Workflow
 
@@ -22,6 +23,7 @@ quorabust-report \
   --artifact-label quorabust-smoke.pkl \
   --eval-csv examples/smoke_pairs.csv \
   --thresholds 0.3,0.5,0.7 \
+  --calibration-bins 5 \
   --out /tmp/quorabust-smoke-model-card.md
 
 quorabust-report \
@@ -39,6 +41,11 @@ Reports include precision, recall, F1, accuracy, and predicted-positive rate at
 selected threshold plus a threshold sweep. Use `--thresholds` to compare operating
 points before choosing one.
 
+Reports also include calibration diagnostics: Brier score, expected calibration error,
+mean predicted probability, mean observed rate, and probability-bin rows. Use
+`--calibration-bins` to control how many bins are printed. Calibration helps decide
+whether probabilities are fit for thresholding or only useful for ranking.
+
 ## Real Evaluation
 
 For comparable numbers, generate the report from a held-out CSV that was not used for
@@ -57,6 +64,7 @@ quorabust-report \
   --artifact-label quorabust-tfidf-v1.pkl \
   --eval-csv data/processed/holdout.csv \
   --thresholds 0.2,0.3,0.4,0.5,0.6,0.7,0.8 \
+  --calibration-bins 10 \
   --out reports/quorabust-tfidf-v1.md
 ```
 
diff --git a/docs/SAMPLE_MODEL_CARD.md b/docs/SAMPLE_MODEL_CARD.md
@@ -1,12 +1,9 @@
 # Quorabust Model Card
 
-> ⚠️ **Smoke demo, not a benchmark.** This card was generated by `quorabust-report`
-> from the tiny synthetic [`examples/smoke_pairs.csv`](../examples/smoke_pairs.csv)
-> (32 rows) — the same fixture CI uses to exercise the train → report path. Its
-> classes are trivially separable, so the scores below describe that toy data and say
-> **nothing** about real-world quality on the Kaggle Quora Question Pairs set. It
-> exists only to show the shape of a generated card. Regenerate with the commands in
-> [Reproduce](#reproduce).
+> Smoke demo, not a benchmark. This card was generated by `quorabust-report` from the
+> tiny synthetic [`examples/smoke_pairs.csv`](../examples/smoke_pairs.csv) fixture. The
+> scores below describe that toy data and say nothing about real-world quality on the
+> Kaggle Quora Question Pairs set. It exists only to show the shape of a generated card.
 
 ## Artifact
 
@@ -29,7 +26,7 @@ Scores pairs of short natural-language questions and returns the probability tha
 | n_eval | 0 |
 | seed | 7 |
 | quorabust_version | 0.3.2 |
-| git_revision | bb7c63ccdf8b22f8ed963dd312013eb2bd6f6719 |
+| git_revision | f86b91a0b6166591d9c366de8ce54d187cb0c3cd |
 | csv_sha256 | 0a7b4ad6c6bb12fd038d5d7673eaced0646a8914c52dbdf502d2476b29ad6132 |
 
 ## Persisted Evaluation
@@ -62,6 +59,26 @@ Scores pairs of short natural-language questions and returns the probability tha
 | actual 0 | 13 | 1 |
 | actual 1 | 1 | 17 |
 
+## Calibration Summary
+
+| metric | value |
+| --- | --- |
+| n_bins | 5 |
+| brier_score | 0.0533 |
+| expected_calibration_error | 0.0916 |
+| mean_predicted_probability | 0.5647 |
+| mean_observed_rate | 0.5625 |
+
+## Calibration Bins
+
+| probability_bin | count | mean_predicted_probability | observed_positive_rate | absolute_error |
+| --- | --- | --- | --- | --- |
+| 0.00-0.20 | 10 | 0.0741 | 0.0000 | 0.0741 |
+| 0.20-0.40 | 4 | 0.3557 | 0.2500 | 0.1057 |
+| 0.40-0.60 | 1 | 0.5334 | 1.0000 | 0.4666 |
+| 0.60-0.80 | 2 | 0.6685 | 0.5000 | 0.1685 |
+| 0.80-1.00 | 15 | 0.9358 | 1.0000 | 0.0642 |
+
 ## Threshold Sweep
 
 | threshold | precision | recall | f1 | accuracy | predicted_positive_rate |
@@ -72,18 +89,33 @@ Scores pairs of short natural-language questions and returns the probability tha
 
 ## Serving Contract
 
-`POST /predict` accepts `question1` and `question2` arrays of equal length and returns `proba_duplicate` in the same order. `GET /metrics` exposes Prometheus counters and latency histograms.
+`POST /predict` accepts `question1` and `question2` arrays of equal length and returns `proba_duplicate`, `is_duplicate`, and `decision_threshold` in the same order. `GET /metrics` exposes Prometheus counters and latency histograms.
 
 ## Caveats
 
 Performance depends on the training data distribution and threshold. Re-run this card on a current holdout set before comparing artifacts.
 
 ## Reproduce
 
+Generated from the checked-in smoke fixture:
+
 ```bash
-pip install -e ".[dev]"
-quorabust-train  --csv examples/smoke_pairs.csv --out /tmp/quorabust-smoke.pkl --eval-fraction 0 --seed 7
-quorabust-report --model /tmp/quorabust-smoke.pkl --artifact-label quorabust-smoke.pkl --eval-csv examples/smoke_pairs.csv --thresholds 0.3,0.5,0.7 --out docs/SAMPLE_MODEL_CARD.md
+quorabust-train \
+  --csv examples/smoke_pairs.csv \
+  --out /tmp/quorabust-smoke.pkl \
+  --metadata-out /tmp/quorabust-smoke.meta.json \
+  --eval-fraction 0 \
+  --seed 7
+
+quorabust-report \
+  --model /tmp/quorabust-smoke.pkl \
+  --artifact-label quorabust-smoke.pkl \
+  --eval-csv examples/smoke_pairs.csv \
+  --thresholds 0.3,0.5,0.7 \
+  --calibration-bins 5 \
+  --out docs/SAMPLE_MODEL_CARD.md
 ```
 
-For a real model card, run the same `train`/`report` commands against the Kaggle Quora Question Pairs CSV instead of the smoke fixture.
+For a real model card, run the same `train` and `report` commands against the Kaggle
+Quora Question Pairs CSV or a labeled customer-domain holdout instead of the smoke
+fixture.
diff --git a/src/quorabust/report.py b/src/quorabust/report.py
@@ -11,6 +11,7 @@
 import pandas as pd
 from sklearn.metrics import (
     accuracy_score,
+    brier_score_loss,
     confusion_matrix,
     f1_score,
     log_loss,
@@ -118,6 +119,54 @@ def _metrics_at_threshold(y: np.ndarray, proba: np.ndarray, threshold: float) ->
     }
 
 
+def calibration_summary(
+    y: np.ndarray,
+    proba: np.ndarray,
+    *,
+    n_bins: int = 10,
+) -> dict[str, Any]:
+    """Summarize whether predicted probabilities match observed positive rates."""
+    if n_bins < 1:
+        raise ValueError("n_bins must be at least 1")
+
+    bins: list[dict[str, Any]] = []
+    total = len(y)
+    expected_calibration_error = 0.0
+    for idx in range(n_bins):
+        lower = idx / n_bins
+        upper = (idx + 1) / n_bins
+        if idx == n_bins - 1:
+            mask = (proba >= lower) & (proba <= upper)
+        else:
+            mask = (proba >= lower) & (proba < upper)
+        count = int(np.sum(mask))
+        if count == 0:
+            continue
+        mean_predicted = float(np.mean(proba[mask]))
+        observed_rate = float(np.mean(y[mask]))
+        absolute_error = abs(mean_predicted - observed_rate)
+        expected_calibration_error += (count / total) * absolute_error
+        bins.append(
+            {
+                "lower": float(lower),
+                "upper": float(upper),
+                "count": count,
+                "mean_predicted_probability": mean_predicted,
+                "observed_positive_rate": observed_rate,
+                "absolute_error": float(absolute_error),
+            }
+        )
+
+    return {
+        "n_bins": int(n_bins),
+        "brier_score": float(brier_score_loss(y, proba)),
+        "expected_calibration_error": float(expected_calibration_error),
+        "mean_predicted_probability": float(np.mean(proba)),
+        "mean_observed_rate": float(np.mean(y)),
+        "bins": bins,
+    }
+
+
 def _holdout_proba(
     builder: Any,
     clf: Any,
@@ -139,6 +188,7 @@ def evaluate_holdout(
     df: pd.DataFrame,
     *,
     threshold: float = 0.5,
+    calibration_bins: int = 10,
 ) -> dict[str, Any]:
     """Evaluate a loaded artifact against a labeled Quora-style dataframe."""
     y, proba = _holdout_proba(builder, clf, df)
@@ -147,6 +197,7 @@ def evaluate_holdout(
         "n": int(len(df)),
         "log_loss": float(log_loss(y, proba, labels=[0, 1])),
         "positive_rate": float(np.mean(y)),
+        "calibration": calibration_summary(y, proba, n_bins=calibration_bins),
         **threshold_metrics,
     }
     if len(np.unique(y)) > 1:
@@ -215,6 +266,7 @@ def render_model_card(
     ]
 
     if holdout_metrics is not None:
+        calibration = holdout_metrics.get("calibration")
         metric_keys = [
             "n",
             "threshold",
@@ -244,6 +296,45 @@ def render_model_card(
                 _markdown_table(["", "predicted 0", "predicted 1"], confusion_rows),
             ]
         )
+        if isinstance(calibration, dict):
+            calibration_rows = [
+                ["n_bins", calibration["n_bins"]],
+                ["brier_score", calibration["brier_score"]],
+                ["expected_calibration_error", calibration["expected_calibration_error"]],
+                ["mean_predicted_probability", calibration["mean_predicted_probability"]],
+                ["mean_observed_rate", calibration["mean_observed_rate"]],
+            ]
+            bin_rows = [
+                [
+                    f"{row['lower']:.2f}-{row['upper']:.2f}",
+                    row["count"],
+                    row["mean_predicted_probability"],
+                    row["observed_positive_rate"],
+                    row["absolute_error"],
+                ]
+                for row in calibration.get("bins", [])
+            ]
+            parts.extend(
+                [
+                    "",
+                    "## Calibration Summary",
+                    "",
+                    _markdown_table(["metric", "value"], calibration_rows),
+                    "",
+                    "## Calibration Bins",
+                    "",
+                    _markdown_table(
+                        [
+                            "probability_bin",
+                            "count",
+                            "mean_predicted_probability",
+                            "observed_positive_rate",
+                            "absolute_error",
+                        ],
+                        bin_rows,
+                    ),
+                ]
+            )
         if sweep_metrics:
             sweep_rows = [
                 [
@@ -282,8 +373,9 @@ def render_model_card(
             "",
             (
                 "`POST /predict` accepts `question1` and `question2` arrays of equal length "
-                "and returns `proba_duplicate` in the same order. `GET /metrics` exposes "
-                "Prometheus counters and latency histograms."
+                "and returns `proba_duplicate`, `is_duplicate`, and `decision_threshold` "
+                "in the same order. `GET /metrics` exposes Prometheus counters and "
+                "latency histograms."
             ),
             "",
             "## Caveats",
@@ -319,7 +411,11 @@ def build_report_payload(
             "predict": "POST /predict",
             "metrics": "GET /metrics",
             "input": {"question1": "list[str]", "question2": "list[str]"},
-            "output": {"proba_duplicate": "list[float]"},
+            "output": {
+                "proba_duplicate": "list[float]",
+                "is_duplicate": "list[bool]",
+                "decision_threshold": "float",
+            },
         },
         "caveats": [
             "Performance depends on the training data distribution and threshold.",
@@ -330,8 +426,11 @@ def build_report_payload(
         payload["holdout_evaluation"] = {
             k: v
             for k, v in holdout_metrics.items()
-            if k not in {"tn", "fp", "fn", "tp"}
+            if k not in {"tn", "fp", "fn", "tp", "calibration"}
         }
+        calibration = holdout_metrics.get("calibration")
+        if calibration is not None:
+            payload["calibration"] = calibration
         payload["confusion_matrix"] = {
             "labels": ["not_duplicate", "duplicate"],
             "actual_0": {
@@ -370,6 +469,12 @@ def main(argv: list[str] | None = None) -> int:
         default="0.3,0.5,0.7",
         help="Comma-separated thresholds for the optional holdout sweep",
     )
+    parser.add_argument(
+        "--calibration-bins",
+        type=int,
+        default=10,
+        help="Number of probability bins for the optional calibration report",
+    )
     parser.add_argument(
         "--artifact-label",
         default=None,
@@ -390,6 +495,9 @@ def main(argv: list[str] | None = None) -> int:
     if not 0.0 < args.threshold < 1.0:
         print("--threshold must be between 0 and 1", file=sys.stderr)
         return 1
+    if args.calibration_bins < 1:
+        print("--calibration-bins must be at least 1", file=sys.stderr)
+        return 1
     try:
         thresholds = _parse_thresholds(args.thresholds)
     except ValueError as exc:
@@ -410,6 +518,7 @@ def main(argv: list[str] | None = None) -> int:
                 clf,
                 eval_df,
                 threshold=args.threshold,
+                calibration_bins=args.calibration_bins,
             )
             sweep_metrics = threshold_sweep(
                 builder,
diff --git a/tests/test_report.py b/tests/test_report.py
