[DOC] Gallery rework: basic examples implementation

CHANGELOG.rst

@@ -26,6 +26,7 @@ under development (0.3.2.dev0)
 - :bdg-danger:`Fix` Fixed unnecessary copy operations of X when only a slice view is needed (:gh:`646` by `Marc Hulcelle`_).
 - :bdg-secondary:`Maint` remove pinned poosh dependency (problem with somato dataset solved) (:gh:`670` by `Joseph Paillard`_).
 - :bdg-secondary:`Maint` remove extra term in variance of X-residual (DOCRT). See  [Reid et al., A Study of Error Variance Estimation in Lasso Regression 2016](https://arxiv.org/pdf/1311.5274) for reference. (:gh:`649` by `Joseph Paillard`_).
+- :bdg-primary:`Doc` example gallery reorganization: added basic "How to get started" examples (:gh:`684` by `Marc Hulcelle`_).
 - :bdg-success:`Feature` add leave-one-covariate-in (LOCI) method (:gh:`679` by `Marc Hulcelle`).
 - :bdg-danger:`Fix` fixed deprecated n_alphas with sklearn LassoCV, as well as deprecated penalty for LogisticRegressionCV (:gh:`690` by `Marc Hulcelle`)
-- :bdg-secondary:`Maint`: Fix conditional sampling test by verifying that sampler produces diverse samples. (:gh:`692` by `Joseph Paillard`_).
+- :bdg-secondary:`Maint` Fix conditional sampling test by verifying that sampler produces diverse samples. (:gh:`692` by `Joseph Paillard`_).

docs/src/model_agnostic_methods/conditional_feature_importance.rst

@@ -12,9 +12,9 @@ unmodified test data—following the same distribution as the training data—
 to its performance when the studied feature is conditionally perturbed. Thus, this approach
 does not require retraining the model.
 
-.. figure:: ../generated/gallery/examples/images/sphx_glr_plot_cfi_001.png
-    :target: ../generated/gallery/examples/plot_cfi.html
-    :align: center
+    #.. figure:: ../generated/gallery/examples/images/sphx_glr_plot_cfi_001.png
+    #    :target: ../generated/gallery/examples/plot_cfi.html
+    #    :align: center
 
 
 Theoretical index

examples/basic_examples/README.rst

@@ -0,0 +1,3 @@
+Getting Started
+===============
+Examples to illustrate the basic usage of HiDimStat.

examples/basic_examples/plot_basic_1_general_pipeline.py

@@ -0,0 +1,143 @@
+"""
+General pipeline to assess feature importance
+==================================================================================
+This example demonstrates how to measure feature importance using hidimstat, and the
+general pipeline of functions to call when doing so.
+The functions used in the following are generic to any of the feature importance assessment
+methods existing in the library. Here, we will use the Conditional Feature Importance
+(CFI) [:footcite:t:`Chamma_NeurIPS2023`] on a simulated regression dataset.
+"""
+
+# %%
+# Loading and preparing the data
+# ------------------------------
+# We begin by simulating a regression dataset with 10 features, 5 of which
+# are in the support set, meaning they contribute to generating the outcome. In this example,
+# we use a simulated dataset to have access to the true support set of features and
+# evaluate how well the different models identify these important features.
+# The data is then split into training and test sets. These sets are used both to fit
+# the predictive models and within the LOCO procedure, which refits models on subsets
+# of features that exclude the feature of interest.
+
+from sklearn.datasets import make_regression
+from sklearn.model_selection import train_test_split
+
+X, y, beta = make_regression(
+    n_samples=300,
+    n_features=10,
+    n_informative=5,
+    random_state=0,
+    coef=True,
+    noise=10.0,
+)
+
+# We convert the coefficients of the data-generating process into a binary array
+# indicating the true support set of features.
+beta = beta != 0
+
+X_train, X_test, y_train, y_test = train_test_split(
+    X,
+    y,
+    test_size=0.2,
+    random_state=0,
+    shuffle=True,
+)
+
+# %%
+# Fitting the model and computing feature importance
+# ------------------------------------------------------
+# To solve the classification task, we use a pipeline that first standardizes the features with StandardScaler,
+# followed by a neural network (MLPClassifier) with one hidden layer of 8 neurons.
+# Before measuring feature importance, we evaluate the estimator's performance by reporting its :math:`R^2` score.
+
+from sklearn.ensemble import RandomForestRegressor
+
+clf = RandomForestRegressor(n_estimators=150, random_state=0)
+
+clf.fit(X_train, y_train)
+y_pred = clf.predict(X_test)
+print(f"Accuracy: {clf.score(X_test, y_test):.3f}")
+
+# %%
+# Next, we use the CFI class to measure feature importance. Here, we use a RidgeCV
+# model to estimate the conditional expectation :math:`\mathbb{E}[X^j | X^{-j}]`.
+# Since this is a regression task, we use mean_squared_error.
+
+from sklearn.linear_model import RidgeCV
+from sklearn.metrics import mean_squared_error
+
+from hidimstat import CFI
+
+cfi = CFI(
+    estimator=clf,
+    loss=mean_squared_error,
+    imputation_model_continuous=RidgeCV(),
+    features_groups={f"Feat {i}": [i] for i in range(X.shape[1])},
+    random_state=0,
+)
+cfi.fit(
+    X_train,
+    y_train,
+)
+importances = cfi.importance(X_test, y_test)
+
+# %%
+# An alternative is to call the `fit_importance` method which conveniently combines both functions.
+# Please keep in mind that both the fit and feature importance will subsequently use the same data.
+
+
+# %%
+# Visualization of CFI feature importance and feature selection
+# ----------------------------------------------------------------
+# Finally, we visualize the importance of each feature using a bar plot
+# thanks to a built-in importance visualization function, and perform
+# feature selection with statistical guarantees.
+
+import matplotlib.pyplot as plt
+import numpy as np
+
+_, ax = plt.subplots(figsize=(6, 3))
+ax = cfi.plot_importance(ax=ax)
+ax.set_xlabel("Feature Importance")
+
+# Since the figure displays importance measures in a descending order,
+# we need to sort betas accordingly:
+
+sorted_beta = beta[np.argsort(importances)[::-1]]
+
+for i, support in enumerate(sorted_beta):
+    if support != 0:
+        ax.axhspan(
+            i - 0.45,
+            i + 0.45,
+            color="tab:olive",
+            alpha=0.3,
+            zorder=-1,
+            label="True Support" if i == 1 else None,
+        )
+ax.legend()
+
+plt.tight_layout()
+plt.show()
+
+
+# %%
+# Next, we can make a function call to select the most important features. This can be done through
+# one of three difference control mechanisms: through p-values, on False Discovery Rate (FDR)
+# or on Family-Wise Error Rate (FWER). Each function returns a boolean mask that indicates us
+# which features to keep based on the selected control mechanism.
+# Here, we will use select importance through p_values and set a standard threshold of 0.05.
+
+selection = cfi.pvalue_selection(threshold_max=0.05)
+important_features = X[:, selection]
+
+# %%
+# This is the basic pipeline for feature importance assessment and feature selection.
+# We demonstrated the general workflow using CFI, but other implemented methods can be used
+# in a similar manner.
+
+
+# %%
+# References
+# ----------
+# .. footbibliography::

examples/basic_examples/plot_basic_2_individual_vs_group.py

@@ -0,0 +1,127 @@
+"""
+Measuring Individual and Group Variable Importance for Classification
+======================================================================
+
+In this example, we show on the Iris dataset how to measure variable importance for
+classification tasks. This time, we use Conditional Feature Importance (CFI) method
+with a Support Vector Classifier (SVC). We start by measuring the importance of
+individual variables and then show how to measure the importance of groups of variables.
+
+To briefly summarize, CFI generates new samples for a feature while preserving the conditional
+distribution over all other features and measures the increase in the loss when predicting (using the same
+full model) on the shuffled data.
+
+"""
+
+# %%
+# Load the iris dataset and add a spurious feature
+# ------------------------------------------------
+# We start by loading the iris dataset as is, and splitting for training and testing.
+
+import numpy as np
+from sklearn.datasets import load_wine
+from sklearn.model_selection import train_test_split
+
+# Define the seeds for the reproducibility of the example
+rng = np.random.default_rng(0)
+
+dataset = load_wine()
+X, y = dataset.data, dataset.target
+feature_names = np.array(dataset.feature_names)
+print(feature_names)
+X_train, X_test, y_train, y_test = train_test_split(
+    X,
+    y,
+    test_size=0.2,
+    random_state=0,
+    shuffle=True,
+)
+
+# %%
+# We use a Multi-Layer Perceptron with one hidden layer of size 100.
+# We fit the estimator, compute the feature importance, and use the
+# built-in importance plot function.
+
+import matplotlib.pyplot as plt
+from sklearn.linear_model import RidgeCV
+from sklearn.metrics import log_loss
+from sklearn.neural_network import MLPClassifier
+from sklearn.pipeline import make_pipeline
+from sklearn.preprocessing import StandardScaler
+
+from hidimstat import CFI
+
+model = make_pipeline(
+    StandardScaler(),
+    MLPClassifier(hidden_layer_sizes=(100), random_state=0, max_iter=500),
+)
+model.fit(X_train, y_train)
+
+cfi = CFI(
+    estimator=model,
+    imputation_model_continuous=RidgeCV(),
+    n_permutations=50,
+    random_state=2,
+    method="predict_proba",
+    loss=log_loss,
+)
+
+cfi.fit(X_train, y_train)
+importance = cfi.importance(X_test, y_test)
+selection = cfi.pvalue_selection(threshold_max=0.05)
+
+_, ax = plt.subplots(figsize=(6, 3))
+ax = cfi.plot_importance(ax=ax)
+ax.set_xlabel("Feature Importance")
+ax.set_yticklabels(feature_names[np.argsort(importance)[::-1]])
+# ax.text(0, 0, f"Selected features: {feature_names[selection]}")
+plt.tight_layout()
+plt.show()
+
+
+# %%
+# Measuring the importance of groups of features
+# ----------------------------------------------
+# In the example above, CFI did not select some features. This is because it
+# measures conditional importance, which is the additional independent information a
+# feature provides knowing all the other features. When features are highly correlated,
+# this additional information decreases, resulting in lower importance rankings. To
+# mitigate this issue, we can group correlated features together and measure the
+# importance of these feature groups. Here, we regroup variables into the following groups:
+# - "acid": `malic_acid` and `proline`
+# - "ash": `ash` and `alcalinity of ash`
+# - "phenols": `total_phenols`, `flavanoids`, `nonflavanoid_phenols`, and `proanthocyanins`
+# - "color": `hue`, and `color_intensity`
+
+features_groups = {
+    "acid": [1, 12],
+    "ash": [2, 3],
+    "phenols": [5, 6, 7, 8],
+    "color": [9, 10],
+}
+
+
+cfi = CFI(
+    estimator=model,
+    imputation_model_continuous=RidgeCV(),
+    n_permutations=50,
+    random_state=2,
+    method="predict_proba",
+    loss=log_loss,
+    features_groups=features_groups,
+)
+
+importance = cfi.fit_importance(X_test, y_test)
+
+_, ax = plt.subplots(figsize=(6, 3))
+ax = cfi.plot_importance(ax=ax)
+ax.set_xlabel("Feature Importance")
+plt.tight_layout()
+plt.show()
+
+# %%
+# Important
+# ---------
+# Feature group analysis is available only for perturbation based method, such as:
+# Permutation Feature Importance (PFI), Conditional Feature Importance (CFI),
+# Leave-one-covariate-in and out (LOCI and LOCO).