docs/source/notebooks/Benchmarking_different_CF_explanation_methods.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "aa1d9cb2",
   "metadata": {},
   "source": [
    "# Benchmarking different CF explanation methods"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c80807c",
   "metadata": {},
   "source": [
    "In this notebook, we show runtimes of different model-agnostic explanation methods. Currently, we support three model-agnostic explanation methods:\n",
    "1. Random-Sampling\n",
    "2. Genetic Algorithm\n",
    "3. Querying a KD tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36dfb4d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import timeit\n",
    "import random\n",
    "\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "import dice_ml\n",
    "from dice_ml.utils import helpers  # helper functions\n",
    "from dice_ml import Dice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9f21b8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dba2138",
   "metadata": {},
   "source": [
    "## Loading dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c1d419f",
   "metadata": {},
   "source": [
    "We use the \"adult\" income dataset from UCI Machine Learning Repository (https://archive.ics.uci.edu/ml/datasets/adult). For demonstration purposes, we transform the data as described in dice_ml.utils.helpers module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9620c49c",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = helpers.load_adult_income_dataset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13014082",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9961bf82",
   "metadata": {},
   "outputs": [],
   "source": [
    "d = dice_ml.Data(dataframe=dataset,\n",
    "                 continuous_features=['age', 'hours_per_week'], outcome_name='income')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae7e84b9",
   "metadata": {},
   "source": [
    "## Training the ML model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ffefbfe",
   "metadata": {},
   "source": [
    "Currently, the genetic algorithm & KD tree methods work with scikit-learn models. Support for Tensorflow 1&2 and Pytorch will be implemented soon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00e43d3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "target = dataset[\"income\"]\n",
    "\n",
    "# Split data into train and test\n",
    "datasetX = dataset.drop(\"income\", axis=1)\n",
    "x_train, x_test, y_train, y_test = train_test_split(datasetX,\n",
    "                                                    target,\n",
    "                                                    test_size=0.2,\n",
    "                                                    random_state=0,\n",
    "                                                    stratify=target)\n",
    "\n",
    "numerical = [\"age\", \"hours_per_week\"]\n",
    "categorical = x_train.columns.difference(numerical)\n",
    "\n",
    "# We create the preprocessing pipelines for both numeric and categorical data.\n",
    "numeric_transformer = Pipeline(\n",
    "    steps=[('scaler', StandardScaler())])\n",
    "\n",
    "categorical_transformer = Pipeline(\n",
    "    steps=[('onehot', OneHotEncoder(handle_unknown='ignore'))])\n",
    "\n",
    "transformations = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('num', numeric_transformer, numerical),\n",
    "        ('cat', categorical_transformer, categorical)])\n",
    "\n",
    "# Append classifier to preprocessing pipeline.\n",
    "# Now we have a full prediction pipeline.\n",
    "clf = Pipeline(steps=[('preprocessor', transformations),\n",
    "                      ('classifier', RandomForestClassifier())])\n",
    "model = clf.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3845b142",
   "metadata": {},
   "outputs": [],
   "source": [
    "m = dice_ml.Model(model=model, backend=\"sklearn\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee61a3fb",
   "metadata": {},
   "source": [
    "## Initialize counterfactual generation methods"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9dc3e16",
   "metadata": {},
   "source": [
    "We now initialize all three counterfactuals generation methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82cfd52d",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp_random = Dice(d, m, method=\"random\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7e17587",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp_genetic = Dice(d, m, method=\"genetic\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3512a70d",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp_KD = Dice(d, m, method=\"kdtree\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b453cbc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_instances = x_train[4:7]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef52d537",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_instances"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b1ae17c",
   "metadata": {},
   "source": [
    "## Generate Counterfactuals"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "953cd3a6",
   "metadata": {},
   "source": [
    "We now generate counterfactuals of desired_class=0 using all three different methods and check the runtime. You can modify the number of loops (```num_loops```), and the number of diverse counterfactuals to generate (```k```). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2acd011",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_loops = 2\n",
    "k = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bfeb5b5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "elapsed_random = 0\n",
    "elapsed_kd = 0\n",
    "elapsed_genetic = 0\n",
    "\n",
    "for _ in range(num_loops):\n",
    "    for q in query_instances:\n",
    "        if q in d.categorical_feature_names:\n",
    "            query_instances.loc[:, q] = \\\n",
    "                [random.choice(dataset[q].values.unique()) for _ in query_instances.index]\n",
    "        else:\n",
    "            query_instances.loc[:, q] = \\\n",
    "                [np.random.uniform(dataset[q].min(), dataset[q].max()) for _ in query_instances.index]\n",
    "\n",
    "    start_time = timeit.default_timer()\n",
    "    dice_exp_random = exp_random.generate_counterfactuals(query_instances, total_CFs=k,\n",
    "                                                          desired_class=0, verbose=False)\n",
    "    elapsed_random += timeit.default_timer() - start_time\n",
    "\n",
    "    start_time = timeit.default_timer()\n",
    "    dice_exp = exp_genetic.generate_counterfactuals(query_instances, total_CFs=k, desired_class=0,\n",
    "                                                    yloss_type=\"hinge_loss\", verbose=False)\n",
    "    elapsed_genetic += timeit.default_timer() - start_time\n",
    "\n",
    "    start_time = timeit.default_timer()\n",
    "    dice_kd = exp_KD.generate_counterfactuals(query_instances, total_CFs=k, desired_class=0,\n",
    "                                              verbose=False)\n",
    "    elapsed_kd += timeit.default_timer() - start_time\n",
    "\n",
    "m_random, s_random = divmod(elapsed_random, 60)\n",
    "print('For Independent random sampling of features: Total time taken to generate %d' % num_loops,\n",
    "      'sets of %d' % k, 'counterfactuals each: %02d' % m_random, 'min %02d' % s_random, 'sec')\n",
    "\n",
    "m_kd, s_kd = divmod(elapsed_kd, 60)\n",
    "print('For querying from a KD tree: Total time taken to generate %d' % num_loops,\n",
    "      'sets of %d' % k, 'counterfactuals each: %02d' % m_kd, 'min %02d' % s_kd, 'sec')\n",
    "\n",
    "m_genetic, s_genetic = divmod(elapsed_genetic, 60)\n",
    "print('For genetic algorithm: Total time taken to generate %d' % num_loops,\n",
    "      'sets of %d' % k, 'counterfactuals each: %02d' % m_genetic, 'min %02d' % s_genetic, 'sec')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}