feat(EstimatorReport): Show permutation at different stages of a pipeline

[auguste-probabl]

Added support for calculating permutation importance at different stages of a pipeline.
One can choose to compute feature importance either at the start of the pipeline or at the end.

Closes #1398

Supersedes #1888

Co-authored-by: waridrox [email protected]

[github-actions]

Coverage Report for skore/

File	Stmts	Miss	Cover	Missing
skore/src/skore
__init__.py	23	0	100%
_config.py	31	0	100%
exceptions.py	4	4	0%	4, 15, 19, 23
skore/src/skore/_sklearn
__init__.py	6	0	100%
_base.py	198	14	92%	45, 58, 127, 130, 183, 186–187, 189–192, 225, 228–229
find_ml_task.py	61	0	100%
types.py	27	1	96%	28
skore/src/skore/_sklearn/_comparison
__init__.py	7	0	100%
feature_importance_accessor.py	35	2	94%	88, 107
metrics_accessor.py	178	3	98%	173, 253, 1215
report.py	107	0	100%
utils.py	54	0	100%
skore/src/skore/_sklearn/_cross_validation
__init__.py	9	0	100%
data_accessor.py	45	3	93%	134, 137, 140
feature_importance_accessor.py	24	0	100%
metrics_accessor.py	182	1	99%	244
report.py	135	1	99%	487
skore/src/skore/_sklearn/_estimator
__init__.py	9	0	100%
data_accessor.py	66	1	98%	82
feature_importance_accessor.py	168	2	98%	251–252
metrics_accessor.py	356	8	97%	200, 202, 209, 300, 369, 373, 388, 423
report.py	165	2	98%	448–449
skore/src/skore/_sklearn/_plot
__init__.py	3	0	100%
base.py	98	6	93%	61–62, 224–226, 230
utils.py	77	0	100%
skore/src/skore/_sklearn/_plot/data
__init__.py	2	0	100%
table_report.py	185	1	99%	706
skore/src/skore/_sklearn/_plot/metrics
__init__.py	6	0	100%
confusion_matrix.py	70	4	94%	92, 100, 122, 230
feature_importance_display.py	67	21	68%	88, 121–122, 124, 142–146, 148–155, 158–160, 162
metrics_summary_display.py	8	0	100%
precision_recall_curve.py	281	5	98%	455, 555, 559, 619, 751
prediction_error.py	227	5	97%	179, 186, 422, 505, 705
roc_curve.py	294	8	97%	387, 510, 515, 616, 621, 625, 694, 834
skore/src/skore/_sklearn/train_test_split
__init__.py	0	0	100%
train_test_split.py	58	0	100%
skore/src/skore/_sklearn/train_test_split/warning
__init__.py	8	0	100%
high_class_imbalance_too_few_examples_warning.py	19	1	94%	83
high_class_imbalance_warning.py	20	0	100%
random_state_unset_warning.py	10	0	100%
shuffle_true_warning.py	9	0	100%
stratify_is_set_warning.py	10	0	100%
time_based_column_warning.py	21	0	100%
train_test_split_warning.py	3	0	100%
skore/src/skore/_utils
__init__.py	6	2	66%	8, 13
_accessor.py	90	3	96%	34, 146, 190
_environment.py	27	1	96%	40
_fixes.py	8	0	100%
_index.py	5	0	100%
_logger.py	22	4	81%	15–17, 19
_measure_time.py	10	0	100%
_parallel.py	38	3	92%	23, 33, 124
_patch.py	13	5	61%	21, 23–24, 35, 37
_progress_bar.py	46	0	100%
_repr_html.py	8	0	100%
_show_versions.py	38	0	100%
_testing.py	55	0	100%
skore/src/skore/project
__init__.py	2	0	100%
project.py	48	0	100%
summary.py	75	1	98%	120
widget.py	187	0	100%
TOTAL	4044	112	97%

Tests	Skipped	Failures	Errors	Time
1101	5 💤	0 ❌	0 🔥	4m 21s ⏱️

[github-actions]

Documentation preview @ 2cbc1f1

[glemaitre]

Since we still don't have the permutation importance across the different reporters, there is not documentation to change in the user guide.

[glemaitre]

It looks good. I would only suggest to amend the example called plot_feature_importance.py where we can demonstate the feature. We have a Ridge model with some feature importance and we can show that we can compute with index 0 and -1.

[auguste-probabl]

amend the example called plot_feature_importance.py where we can demonstate the feature. We have a Ridge model with some feature importance and we can show that we can compute with index 0 and -1.

TODO: I found a bug while doing this, at a certain step the permutation computation fails because it doesn't like sparse matrices. Fixed.

Now I need to figure out how to showcase the new feature in the example, hopefully in a way that doesn't disrupt the flow.

[glemaitre]

Here it comes ;). Sorry for the delay.

[glemaitre]

With those changes, you are going to cover the two missing line and it is a possible pipeline that we should be supporting.

[auguste-probabl]

Now it's

skore/skore/src/skore/_sklearn/_estimator/feature_importance_accessor.py

Line 632 in 612dee8

feature_names = estimator.feature_names_in_

that is not covered

[glemaitre]

Just the last comment. Otherwise, LGTM.