feat: Add a threshold parameter to the confusion_matrix display

[GaetandeCast]

Closes #2112.

With this PR, for binary classification, calling report.metrics.confusion_matrix(threshold=True) will compute and store the confusion matrices for all thresholds of the decision function of the classifier. They can then be plotted with .plot(threshold_value=x) and accessed via .frame().

This makes use of the new scikit-learn function confusion_matrix_at_thresholds, available in 1.8 and back-ported for earlier versions.

The storage structure extends what we converged towards in #2165 : a long format dataframe with one cell of one matrix per row. Columns are the raw count number, all three possible normalized values, the threshold value, and the true and predicted labels.

The default threshold value is 0.5. We could add an "auto" option to select the "best" threshold if we find a satisfactory universal metric to define what "best" means (balanced accuracy may not be desirable for instance, as argued here).

[github-actions]

Coverage Report for skore/

File	Stmts	Miss	Cover	Missing
skore/src/skore
__init__.py	24	0	100%
_config.py	31	0	100%
exceptions.py	4	4	0%	4, 15, 19, 23
skore/src/skore/_sklearn
__init__.py	6	0	100%
_base.py	199	14	92%	46, 59, 128, 131, 184, 187–188, 190–193, 226, 229–230
find_ml_task.py	61	0	100%
types.py	29	1	96%	30
skore/src/skore/_sklearn/_comparison
__init__.py	7	0	100%
feature_importance_accessor.py	35	2	94%	93, 117
metrics_accessor.py	179	3	98%	169, 249, 1212
report.py	108	0	100%
utils.py	57	0	100%
skore/src/skore/_sklearn/_cross_validation
__init__.py	9	0	100%
data_accessor.py	45	3	93%	134, 137, 140
feature_importance_accessor.py	24	0	100%
metrics_accessor.py	183	1	99%	242
report.py	136	1	99%	490
skore/src/skore/_sklearn/_estimator
__init__.py	9	0	100%
data_accessor.py	66	1	98%	82
feature_importance_accessor.py	168	2	98%	258–259
metrics_accessor.py	386	6	98%	329, 398, 402, 417, 452, 2137
report.py	166	2	98%	449–450
skore/src/skore/_sklearn/_plot
__init__.py	3	0	100%
base.py	106	6	94%	61–62, 247–249, 253
utils.py	77	0	100%
skore/src/skore/_sklearn/_plot/data
__init__.py	2	0	100%
table_report.py	185	1	99%	706
skore/src/skore/_sklearn/_plot/metrics
__init__.py	6	0	100%
confusion_matrix.py	92	0	100%
feature_importance_coefficients_display.py	71	21	70%	116–119, 121, 140, 146–152, 155, 161–165, 170–171
metrics_summary_display.py	8	0	100%
precision_recall_curve.py	301	6	98%	242, 535, 635, 639, 699, 831
prediction_error.py	233	5	97%	179, 186, 423, 506, 706
roc_curve.py	314	9	97%	263, 455, 578, 583, 684, 689, 693, 762, 902
skore/src/skore/_sklearn/train_test_split
__init__.py	0	0	100%
train_test_split.py	58	0	100%
skore/src/skore/_sklearn/train_test_split/warning
__init__.py	8	0	100%
high_class_imbalance_too_few_examples_warning.py	19	1	94%	83
high_class_imbalance_warning.py	20	0	100%
random_state_unset_warning.py	10	0	100%
shuffle_true_warning.py	9	0	100%
stratify_is_set_warning.py	10	0	100%
time_based_column_warning.py	21	0	100%
train_test_split_warning.py	3	0	100%
skore/src/skore/_utils
__init__.py	6	2	66%	8, 13
_accessor.py	90	3	96%	34, 146, 190
_cache.py	23	0	100%
_environment.py	27	1	96%	40
_fixes.py	8	0	100%
_index.py	5	0	100%
_logger.py	22	4	81%	15–17, 19
_measure_time.py	10	0	100%
_parallel.py	38	3	92%	23, 33, 124
_patch.py	21	12	42%	30, 35–39, 42–43, 46–47, 58, 60
_progress_bar.py	46	0	100%
_repr_html.py	8	0	100%
_show_versions.py	38	0	100%
_testing.py	56	0	100%
skore/src/skore/project
__init__.py	2	0	100%
project.py	48	0	100%
summary.py	75	1	98%	120
widget.py	187	0	100%
TOTAL	4198	115	97%

Tests	Skipped	Failures	Errors	Time
1145	5 💤	0 ❌	0 🔥	4m 1s ⏱️

[github-actions]

Documentation preview @ 383cb64

[glemaitre]

It is a first pass on the API design. I'm overlooking the tests since that the changes will have an impact and we can iterate already on the API before to take care about the test.

[GaetandeCast]

Hi @glemaitre, thanks for the review. I have have implemented the requested changes. I will wait until we are satisfied with the API to update the tests and the doc example. So until then, the CI will be red.

[GaetandeCast]

@glemaitre I did changes suggested orally:

• add is an asterisk on the positive label and a legend explaining it
• the decision threshold is now on a new line in the title
• mention "Decision threshold: 0.5" in the title when no threshold was explicitly chosen
• describe the usage and utility of the threshold in the docstring of plot() and frame().

Here is how the display.plot() looks like now:

Generated with:

cm_display = report.metrics.confusion_matrix(pos_label="disallowed")
cm_display.plot(threshold_value=0.3)
plt.show()