feat(model): persist selected decision threshold

Author: AlisinaDevelo

CHANGELOG.md

@@ -14,6 +14,7 @@
 - Thresholded `/predict` decisions with artifact/env defaults and per-request override.
 - Enterprise product-positioning guide for semantic matching use cases and production gaps.
 - Calibration summary and probability-bin diagnostics in `quorabust-report`.
+- Holdout-selected `decision_threshold` metadata from `quorabust-train`.
 
 ### Fixed
 - Capped NumPy below 2.5 so Mypy can parse dependency stubs with the configured

README.md

@@ -66,7 +66,7 @@ quorabust-train --csv data/raw/train.csv --out models/quorabust.pkl
 python -m quorabust --csv data/raw/train.csv --out models/quorabust.pkl   # equivalent
 ```
 
-Options: `--max-rows N`, `--eval-fraction 0.1` (default), `--eval-fraction 0` to train on all rows without a holdout, `--seed`, `--feature-backend {tfidf,embedding}`, `--embedding-model …`, `--registry-dir` (JSONL registry), `--metadata-out` (JSON sidecar for reviewing artifact lineage without loading the pickle).
+Options: `--max-rows N`, `--eval-fraction 0.1` (default), `--eval-fraction 0` to train on all rows without a holdout, `--seed`, `--feature-backend {tfidf,embedding}`, `--embedding-model …`, `--thresholds`, `--threshold-metric {accuracy,precision,recall,f1}` for holdout-based decision-threshold selection, `--registry-dir` (JSONL registry), `--metadata-out` (JSON sidecar for reviewing artifact lineage without loading the pickle).
 
 ### Generate a model card
 
@@ -103,7 +103,7 @@ quorabust-serve --host 0.0.0.0 --port 8000
 # optional second artifact: export QUORABUST_MODEL_B=models/other.pkl
 ```
 
-`GET /metrics` exposes Prometheus text; `POST /predict` accepts `{"question1":[...],"question2":[...]}` and optional header `X-Quorabust-Variant: b`. Responses include `proba_duplicate`, thresholded `is_duplicate`, and `decision_threshold`. Add `?threshold=0.7` to override the duplicate cutoff for one request, or set artifact metadata `decision_threshold` / `QUORABUST_DECISION_THRESHOLD` for the default. Add `?explain=true` to return per-pair input feature values. Interactive docs: **`/docs`**. Load testing: [docs/LOAD_TESTING.md](docs/LOAD_TESTING.md). Grafana: [docs/GRAFANA.md](docs/GRAFANA.md).
+`GET /metrics` exposes Prometheus text; `POST /predict` accepts `{"question1":[...],"question2":[...]}` and optional header `X-Quorabust-Variant: b`. Responses include `proba_duplicate`, thresholded `is_duplicate`, and `decision_threshold`. Add `?threshold=0.7` to override the duplicate cutoff for one request; otherwise serving uses the holdout-selected artifact `decision_threshold` when present, then `QUORABUST_DECISION_THRESHOLD`, then `0.5`. Add `?explain=true` to return per-pair input feature values. Interactive docs: **`/docs`**. Load testing: [docs/LOAD_TESTING.md](docs/LOAD_TESTING.md). Grafana: [docs/GRAFANA.md](docs/GRAFANA.md).
 `GET /models` returns allowlisted metadata for loaded variants without leaking local
 artifact paths or training CSV paths.
 

docs/ENTERPRISE.md

@@ -14,7 +14,10 @@
 
 ## Lineage and artifacts
 
-Training writes `csv_sha256`, `git_revision`, `quorabust_version`, `feature_schema`, `reference_feature_means` (for drift checks), and metric fields into the pickle `meta` dict. Treat `.pkl` files as **trusted** (pickle); load only from controlled storage.
+Training writes `csv_sha256`, `git_revision`, `quorabust_version`, `feature_schema`,
+`reference_feature_means` (for drift checks), holdout-selected `decision_threshold`
+when an eval split exists, and metric fields into the pickle `meta` dict. Treat `.pkl`
+files as **trusted** (pickle); load only from controlled storage.
 
 Use `quorabust-train --metadata-out models/quorabust.meta.json` to write the same
 lineage and metric metadata as JSON. Reviewers and release tooling can inspect that
@@ -26,8 +29,9 @@ prefer a non-pickle format such as `skops` or ONNX in a future release.
 
 - **`quorabust-serve`**: FastAPI with `/health`, `/ready`, `/predict`, `/metrics` (Prometheus). Configure **`QUORABUST_MODEL_PATH`** and optional **`QUORABUST_MODEL_B`** for A/B; clients may send **`X-Quorabust-Variant: b`**.
 - **Decisioning**: `/predict` returns both `proba_duplicate` and thresholded
-  `is_duplicate`. Clients can pass `?threshold=0.7`; otherwise serving uses artifact
-  metadata `decision_threshold`, then `QUORABUST_DECISION_THRESHOLD`, then `0.5`.
+  `is_duplicate`. Clients can pass `?threshold=0.7`; otherwise serving uses the
+  holdout-selected artifact metadata `decision_threshold`, then
+  `QUORABUST_DECISION_THRESHOLD`, then `0.5`.
 - Wire ingress timeouts and autoscaling to your **latency** SLO using the histogram in `/metrics`. See [LOAD_TESTING.md](LOAD_TESTING.md) for k6 and [GRAFANA.md](GRAFANA.md) for a starter dashboard JSON.
 
 ## Scale and NLP

docs/PRODUCT.md

@@ -20,6 +20,7 @@ It does this with:
 - Reproducible offline training.
 - A deployable FastAPI scoring service.
 - Thresholded decisions, not only raw probabilities.
+- Holdout-selected serving thresholds persisted in artifact metadata.
 - Model-card reporting with threshold sweeps.
 - Artifact metadata, JSON sidecars, and safe public model metadata.
 - Prometheus metrics, A/B model routing, drift helpers, and load-test assets.
@@ -54,7 +55,7 @@ dataset or customer-domain labels before making performance claims.
 Decision threshold precedence:
 
 1. Request query parameter: `?threshold=0.7`.
-2. Artifact metadata: `decision_threshold`.
+2. Holdout-selected artifact metadata: `decision_threshold`.
 3. Environment: `QUORABUST_DECISION_THRESHOLD`.
 4. Default: `0.5`.
 

docs/REPORTING.md

@@ -57,6 +57,8 @@ quorabust-train \
   --out models/quorabust.pkl \
   --metadata-out models/quorabust.meta.json \
   --eval-fraction 0.1 \
+  --thresholds 0.2,0.3,0.4,0.5,0.6,0.7,0.8 \
+  --threshold-metric f1 \
   --seed 42
 
 quorabust-report \
@@ -75,6 +77,11 @@ threshold.
 Use `--format json` when you want CI or release tooling to compare metrics without
 scraping Markdown.
 
+When `quorabust-train` has a holdout split, it stores a selected `decision_threshold` in
+artifact metadata. The threshold is chosen from `--thresholds` by maximizing
+`--threshold-metric` (default `f1`). Serving uses that artifact threshold unless the
+request overrides it with `?threshold=...`.
+
 ## Sample model card
 
 A checked-in example produced from the [`examples/smoke_pairs.csv`](../examples/smoke_pairs.csv)

src/quorabust/cli.py

@@ -10,7 +10,12 @@
 
 from quorabust.drift import feature_means_from_matrix
 from quorabust.lineage import git_revision, sha256_file
-from quorabust.model import eval_classification_metrics, train_duplicate_classifier
+from quorabust.model import (
+    eval_classification_metrics,
+    predict_proba_duplicate,
+    select_decision_threshold,
+    train_duplicate_classifier,
+)
 from quorabust.persist import save_classifier, save_metadata_sidecar
 from quorabust.registry import append_model_record
 
@@ -32,6 +37,24 @@ def _load_quora_csv(path: Path) -> pd.DataFrame:
     return df
 
 
+def _parse_thresholds(raw: str) -> list[float]:
+    thresholds: list[float] = []
+    for part in raw.split(","):
+        value = part.strip()
+        if not value:
+            continue
+        try:
+            threshold = float(value)
+        except ValueError as exc:
+            raise ValueError(f"Invalid threshold: {value}") from exc
+        if not 0.0 < threshold < 1.0:
+            raise ValueError("thresholds must be between 0 and 1")
+        thresholds.append(threshold)
+    if not thresholds:
+        raise ValueError("at least one threshold is required")
+    return thresholds
+
+
 def main(argv: list[str] | None = None) -> int:
     p = argparse.ArgumentParser(
         description="Train Quorabust duplicate classifier from a Quora-style CSV.",
@@ -79,11 +102,27 @@ def main(argv: list[str] | None = None) -> int:
         default=None,
         help="If set, write artifact metadata JSON here without requiring pickle loading",
     )
+    p.add_argument(
+        "--thresholds",
+        default="0.2,0.3,0.4,0.5,0.6,0.7,0.8",
+        help="Comma-separated candidate thresholds for holdout-based decision selection",
+    )
+    p.add_argument(
+        "--threshold-metric",
+        choices=["accuracy", "precision", "recall", "f1"],
+        default="f1",
+        help="Metric to optimize when selecting a decision threshold from the holdout",
+    )
     args = p.parse_args(argv)
 
     if not args.csv.is_file():
         print(f"File not found: {args.csv}", file=sys.stderr)
         return 1
+    try:
+        threshold_candidates = _parse_thresholds(args.thresholds)
+    except ValueError as e:
+        print(e, file=sys.stderr)
+        return 1
 
     try:
         df = _load_quora_csv(args.csv)
@@ -135,6 +174,26 @@ def main(argv: list[str] | None = None) -> int:
     m = eval_classification_metrics(builder, clf, eval_target)
     for k, v in m.items():
         meta[f"eval_{k}"] = v
+    if eval_df is not None:
+        y_eval = eval_df["is_duplicate"].astype(int).to_numpy()
+        p_eval = predict_proba_duplicate(
+            builder,
+            clf,
+            eval_df["question1"].astype(str).tolist(),
+            eval_df["question2"].astype(str).tolist(),
+        )[:, 1]
+        selected_threshold = select_decision_threshold(
+            y_eval,
+            p_eval,
+            thresholds=threshold_candidates,
+            optimize_for=args.threshold_metric,
+        )
+        meta["decision_threshold"] = selected_threshold["threshold"]
+        meta["decision_threshold_source"] = "eval_holdout"
+        meta["decision_threshold_metric"] = args.threshold_metric
+        meta["decision_threshold_metrics"] = {
+            k: v for k, v in selected_threshold.items() if k != "threshold"
+        }
     print(
         "metrics: "
         + ", ".join(f"{k}={v:.4f}" for k, v in sorted(m.items())),
@@ -155,6 +214,8 @@ def main(argv: list[str] | None = None) -> int:
                 "feature_backend": args.feature_backend,
                 "git_revision": meta.get("git_revision"),
                 "quorabust_version": meta.get("quorabust_version"),
+                "decision_threshold": meta.get("decision_threshold"),
+                "decision_threshold_metric": meta.get("decision_threshold_metric"),
                 "eval_metrics": {k: meta[k] for k in meta if k.startswith("eval_")},
             },
         )

src/quorabust/model.py

@@ -4,7 +4,14 @@
 
 import numpy as np
 import pandas as pd
-from sklearn.metrics import accuracy_score, log_loss, roc_auc_score
+from sklearn.metrics import (
+    accuracy_score,
+    f1_score,
+    log_loss,
+    precision_score,
+    recall_score,
+    roc_auc_score,
+)
 from xgboost import XGBClassifier
 
 from quorabust.features import PairFeatureBuilder
@@ -89,7 +96,7 @@ def eval_log_loss(
     X = builder.transform_frame(df, col_q1=col_q1, col_q2=col_q2)
     y = df[label_col].astype(int).to_numpy()
     proba = clf.predict_proba(X)[:, 1]
-    return float(log_loss(y, proba))
+    return float(log_loss(y, proba, labels=[0, 1]))
 
 
 def eval_classification_metrics(
@@ -107,8 +114,52 @@ def eval_classification_metrics(
     y_hat = (proba >= 0.5).astype(int)
     out: dict[str, float] = {
         "accuracy": float(accuracy_score(y, y_hat)),
-        "log_loss": float(log_loss(y, proba)),
+        "log_loss": float(log_loss(y, proba, labels=[0, 1])),
     }
     if len(np.unique(y)) > 1:
         out["roc_auc"] = float(roc_auc_score(y, proba))
     return out
+
+
+def select_decision_threshold(
+    y: np.ndarray,
+    proba: np.ndarray,
+    *,
+    thresholds: list[float] | None = None,
+    optimize_for: str = "f1",
+) -> dict[str, float]:
+    """Choose a probability threshold from labeled probabilities."""
+    candidates = thresholds or [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]
+    if optimize_for not in {"accuracy", "precision", "recall", "f1"}:
+        raise ValueError("optimize_for must be one of: accuracy, precision, recall, f1")
+    if not candidates:
+        raise ValueError("at least one threshold is required")
+
+    best: dict[str, float] | None = None
+    for threshold in candidates:
+        if not 0.0 < threshold < 1.0:
+            raise ValueError("thresholds must be between 0 and 1")
+        pred = (proba >= threshold).astype(int)
+        row = {
+            "threshold": float(threshold),
+            "accuracy": float(accuracy_score(y, pred)),
+            "precision": float(precision_score(y, pred, zero_division=0)),
+            "recall": float(recall_score(y, pred, zero_division=0)),
+            "f1": float(f1_score(y, pred, zero_division=0)),
+        }
+        if best is None:
+            best = row
+            continue
+        current_key = (row[optimize_for], row["f1"], row["accuracy"], -abs(row["threshold"] - 0.5))
+        best_key = (
+            best[optimize_for],
+            best["f1"],
+            best["accuracy"],
+            -abs(best["threshold"] - 0.5),
+        )
+        if current_key > best_key:
+            best = row
+
+    if best is None:
+        raise ValueError("at least one threshold is required")
+    return best

src/quorabust/serve.py

@@ -141,6 +141,10 @@ class PredictOut(BaseModel):
     "git_revision",
     "csv_sha256",
     "reference_feature_means",
+    "decision_threshold",
+    "decision_threshold_source",
+    "decision_threshold_metric",
+    "decision_threshold_metrics",
 }
 
 

tests/test_cli.py

@@ -52,6 +52,55 @@ def test_cli_writes_metadata_sidecar(tmp_path):
     assert "eval_accuracy" in payload
 
 
+def test_cli_persists_holdout_decision_threshold(tmp_path):
+    csv = tmp_path / "train.csv"
+    _write_synthetic_csv(csv, n=80)
+    out = tmp_path / "model.pkl"
+    meta = tmp_path / "model.meta.json"
+
+    assert (
+        main(
+            [
+                "--csv",
+                str(csv),
+                "--out",
+                str(out),
+                "--metadata-out",
+                str(meta),
+                "--thresholds",
+                "0.3,0.5,0.7",
+                "--threshold-metric",
+                "f1",
+            ]
+        )
+        == 0
+    )
+
+    payload = json.loads(meta.read_text(encoding="utf-8"))
+    assert payload["decision_threshold"] in {0.3, 0.5, 0.7}
+    assert payload["decision_threshold_source"] == "eval_holdout"
+    assert payload["decision_threshold_metric"] == "f1"
+    assert "f1" in payload["decision_threshold_metrics"]
+
+
+def test_cli_rejects_bad_threshold_grid(tmp_path):
+    csv = tmp_path / "train.csv"
+    _write_synthetic_csv(csv)
+    assert (
+        main(
+            [
+                "--csv",
+                str(csv),
+                "--out",
+                str(tmp_path / "model.pkl"),
+                "--thresholds",
+                "0.2,nope",
+            ]
+        )
+        == 1
+    )
+
+
 def test_cli_rejects_bad_columns(tmp_path):
     csv = tmp_path / "bad.csv"
     pd.DataFrame({"a": [1]}).to_csv(csv, index=False)

tests/test_metrics.py

@@ -1,6 +1,10 @@
 import pandas as pd
 
-from quorabust.model import eval_classification_metrics, train_duplicate_classifier
+from quorabust.model import (
+    eval_classification_metrics,
+    select_decision_threshold,
+    train_duplicate_classifier,
+)
 
 
 def test_eval_classification_metrics_has_auc():
@@ -15,3 +19,31 @@ def test_eval_classification_metrics_has_auc():
     m = eval_classification_metrics(b, clf, df)
     assert "accuracy" in m and "log_loss" in m and "roc_auc" in m
     assert 0.0 <= m["roc_auc"] <= 1.0
+
+
+def test_eval_classification_metrics_handles_one_class_eval():
+    df = pd.DataFrame(
+        {
+            "question1": [f"q1 {i}" for i in range(20)],
+            "question2": [f"q2 {i % 4}" for i in range(20)],
+            "is_duplicate": [i % 2 for i in range(20)],
+        }
+    )
+    b, clf = train_duplicate_classifier(df, xgb_params={"n_estimators": 20, "max_depth": 3})
+    one_class = df[df["is_duplicate"] == 1].head(5).copy()
+    m = eval_classification_metrics(b, clf, one_class)
+    assert "accuracy" in m and "log_loss" in m
+    assert "roc_auc" not in m
+
+
+def test_select_decision_threshold_optimizes_requested_metric():
+    y = pd.Series([0, 0, 1, 1]).to_numpy()
+    proba = pd.Series([0.1, 0.4, 0.6, 0.9]).to_numpy()
+    selected = select_decision_threshold(
+        y,
+        proba,
+        thresholds=[0.3, 0.5, 0.7],
+        optimize_for="f1",
+    )
+    assert selected["threshold"] == 0.5
+    assert selected["f1"] == 1.0