MAINT Replace R^2 with MAE in exercise M3.02

notebooks/parameter_tuning_ex_03.ipynb

@@ -52,8 +52,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Use `RandomizedSearchCV` with `n_iter=20` to find the best set of\n",
-    "hyperparameters by tuning the following parameters of the `model`:\n",
+    "Use `RandomizedSearchCV` with `n_iter=20` and\n",
+    "`scoring=\"neg_mean_absolute_error\"` to tune the following hyperparameters\n",
+    "of the `model`:\n",
     "\n",
     "- the parameter `n_neighbors` of the `KNeighborsRegressor` with values\n",
     "  `np.logspace(0, 3, num=10).astype(np.int32)`;\n",
@@ -62,6 +63,11 @@
     "- the parameter `with_std` of the `StandardScaler` with possible values `True`\n",
     "  or `False`.\n",
     "\n",
+    "The `scoring` function is expected to return higher values for better models,\n",
+    "since grid/random search objects **maximize** it. Because of that, error\n",
+    "metrics like `mean_absolute_error` must be negated (using the `neg_` prefix)\n",
+    "to work correctly (remember lower errors represent better models).\n",
+    "\n",
     "Notice that in the notebook \"Hyperparameter tuning by randomized-search\" we\n",
     "pass distributions to be sampled by the `RandomizedSearchCV`. In this case we\n",
     "define a fixed grid of hyperparameters to be explored. Using a `GridSearchCV`\n",

notebooks/parameter_tuning_sol_03.ipynb

@@ -58,8 +58,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Use `RandomizedSearchCV` with `n_iter=20` to find the best set of\n",
-    "hyperparameters by tuning the following parameters of the `model`:\n",
+    "Use `RandomizedSearchCV` with `n_iter=20` and\n",
+    "`scoring=\"neg_mean_absolute_error\"` to tune the following hyperparameters\n",
+    "of the `model`:\n",
     "\n",
     "- the parameter `n_neighbors` of the `KNeighborsRegressor` with values\n",
     "  `np.logspace(0, 3, num=10).astype(np.int32)`;\n",
@@ -68,6 +69,11 @@
     "- the parameter `with_std` of the `StandardScaler` with possible values `True`\n",
     "  or `False`.\n",
     "\n",
+    "The `scoring` function is expected to return higher values for better models,\n",
+    "since grid/random search objects **maximize** it. Because of that, error\n",
+    "metrics like `mean_absolute_error` must be negated (using the `neg_` prefix)\n",
+    "to work correctly (remember lower errors represent better models).\n",
+    "\n",
     "Notice that in the notebook \"Hyperparameter tuning by randomized-search\" we\n",
     "pass distributions to be sampled by the `RandomizedSearchCV`. In this case we\n",
     "define a fixed grid of hyperparameters to be explored. Using a `GridSearchCV`\n",
@@ -103,6 +109,7 @@
     "model_random_search = RandomizedSearchCV(\n",
     "    model,\n",
     "    param_distributions=param_distributions,\n",
+    "    scoring=\"neg_mean_absolute_error\",\n",
     "    n_iter=20,\n",
     "    n_jobs=2,\n",
     "    verbose=1,\n",
@@ -139,6 +146,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
+    "lines_to_next_cell": 0,
     "tags": [
      "solution"
     ]
@@ -150,6 +158,27 @@
     "cv_results = pd.DataFrame(model_random_search.cv_results_)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As we used `neg_mean_absolute_error` as score metric, we should multiply the\n",
+    "score results with minus 1 to get mean absolute error values:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "solution"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "cv_results[\"mean_test_score\"] *= -1"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {
@@ -181,7 +210,7 @@
     "\n",
     "cv_results = cv_results.rename(columns=column_name_mapping)\n",
     "cv_results = cv_results[column_name_mapping.values()].sort_values(\n",
-    "    \"mean test score\", ascending=False\n",
+    "    \"mean test score\"\n",
     ")"
    ]
   },
@@ -249,7 +278,7 @@
     "holding on any axis of the parallel coordinate plot. You can then slide (move)\n",
     "the range selection and cross two selections to see the intersections.\n",
     "\n",
-    "Selecting the best performing models (i.e. above R2 score of ~0.68), we\n",
+    "Selecting the best performing models (i.e. below MEA score of ~47 k$), we\n",
     "observe that **in this case**:\n",
     "\n",
     "- scaling the data is important. All the best performing models use scaled\n",

python_scripts/parameter_tuning_ex_03.py

@@ -40,8 +40,9 @@
 # Write your code here.
 
 # %% [markdown]
-# Use `RandomizedSearchCV` with `n_iter=20` to find the best set of
-# hyperparameters by tuning the following parameters of the `model`:
+# Use `RandomizedSearchCV` with `n_iter=20` and
+# `scoring="neg_mean_absolute_error"` to tune the following hyperparameters
+# of the `model`:
 #
 # - the parameter `n_neighbors` of the `KNeighborsRegressor` with values
 #   `np.logspace(0, 3, num=10).astype(np.int32)`;
@@ -50,6 +51,11 @@
 # - the parameter `with_std` of the `StandardScaler` with possible values `True`
 #   or `False`.
 #
+# The `scoring` function is expected to return higher values for better models,
+# since grid/random search objects **maximize** it. Because of that, error
+# metrics like `mean_absolute_error` must be negated (using the `neg_` prefix)
+# to work correctly (remember lower errors represent better models).
+#
 # Notice that in the notebook "Hyperparameter tuning by randomized-search" we
 # pass distributions to be sampled by the `RandomizedSearchCV`. In this case we
 # define a fixed grid of hyperparameters to be explored. Using a `GridSearchCV`

python_scripts/parameter_tuning_sol_03.py

@@ -40,8 +40,9 @@
 model = make_pipeline(scaler, KNeighborsRegressor())
 
 # %% [markdown]
-# Use `RandomizedSearchCV` with `n_iter=20` to find the best set of
-# hyperparameters by tuning the following parameters of the `model`:
+# Use `RandomizedSearchCV` with `n_iter=20` and
+# `scoring="neg_mean_absolute_error"` to tune the following hyperparameters
+# of the `model`:
 #
 # - the parameter `n_neighbors` of the `KNeighborsRegressor` with values
 #   `np.logspace(0, 3, num=10).astype(np.int32)`;
@@ -50,6 +51,11 @@
 # - the parameter `with_std` of the `StandardScaler` with possible values `True`
 #   or `False`.
 #
+# The `scoring` function is expected to return higher values for better models,
+# since grid/random search objects **maximize** it. Because of that, error
+# metrics like `mean_absolute_error` must be negated (using the `neg_` prefix)
+# to work correctly (remember lower errors represent better models).
+#
 # Notice that in the notebook "Hyperparameter tuning by randomized-search" we
 # pass distributions to be sampled by the `RandomizedSearchCV`. In this case we
 # define a fixed grid of hyperparameters to be explored. Using a `GridSearchCV`
@@ -79,6 +85,7 @@
 model_random_search = RandomizedSearchCV(
     model,
     param_distributions=param_distributions,
+    scoring="neg_mean_absolute_error",
     n_iter=20,
     n_jobs=2,
     verbose=1,
@@ -107,6 +114,13 @@
 
 cv_results = pd.DataFrame(model_random_search.cv_results_)
 
+# %% [markdown] tags=["solution"]
+# As we used `neg_mean_absolute_error` as score metric, we should multiply the
+# score results with minus 1 to get mean absolute error values:
+
+# %% tags=["solution"]
+cv_results["mean_test_score"] *= -1
+
 # %% [markdown] tags=["solution"]
 # To simplify the axis of the plot, we rename the column of the dataframe and
 # only select the mean test score and the value of the hyperparameters.
@@ -121,7 +135,7 @@
 
 cv_results = cv_results.rename(columns=column_name_mapping)
 cv_results = cv_results[column_name_mapping.values()].sort_values(
-    "mean test score", ascending=False
+    "mean test score"
 )
 
 # %% [markdown] tags=["solution"]
@@ -153,7 +167,7 @@
 # holding on any axis of the parallel coordinate plot. You can then slide (move)
 # the range selection and cross two selections to see the intersections.
 #
-# Selecting the best performing models (i.e. above R2 score of ~0.68), we
+# Selecting the best performing models (i.e. below MEA score of ~47 k$), we
 # observe that **in this case**:
 #
 # - scaling the data is important. All the best performing models use scaled