Applied ml gallery

mlr-org/_freeze/gallery/appliedml/2025-06-02-adv-perf-eval-calibration-mlr3/index/execute-results/html.json

@@ -0,0 +1,17 @@
+{
+  "hash": "db45ba7742a6592491d6f2766b5431aa",
+  "result": {
+    "engine": "knitr",
+    "markdown": "---\ntitle: Calibration with mlr3\ngroup: Advanced Performance Evaluation\ncategories:\n  - calibration\nauthor:\n  - name: Giuseppe Casalicchio\n  - name: Essential Data Science Training GmbH\n    url: https://www.essentialds.de\ndescription: |\n  Learn the basics of `tidymodels` for supervised learning, assess if a model is well-calibrated, and calibrate it with `mlr3`.\ndate: \"\"\nparams:\n  showsolution: true\n  base64encode: true\nlisting: false\nsearch: false\nformat:\n  html:\n    filters:\n      - ../../b64_solution.lua\n---\n\n\n\n\n\n::: {.cell layout-align=\"center\"}\n\n:::\n\n```{=html}\n<script>\nconst correctHash = \"5267a9bd3d0c509f246d69df11d9820c28d6abc8e08094250c5c5532bb757bb1\";   // value injected by knitr\n\n/* ---------- reusable helper ---------- */\nfunction b64DecodeUtf8(b64) {\n  // 1) atob  -> binary-string   (bytes 0…255)\n  // 2) map   -> Uint8Array      (array of bytes)\n  // 3) TextDecoder('utf-8')     -> real JS string\n  const bytes = Uint8Array.from(atob(b64), c => c.charCodeAt(0));\n  return new TextDecoder('utf-8').decode(bytes);\n}\n\nasync function sha256(txt) {\n  const buf = await crypto.subtle.digest('SHA-256',\n                 new TextEncoder().encode(txt));\n  return Array.from(new Uint8Array(buf))\n              .map(b => b.toString(16).padStart(2, '0')).join('');\n}\n\nasync function unlockOne(btn) {\n  const pass = prompt(\"Password:\");\n  if (!pass) return;\n  if (await sha256(pass) !== correctHash) {\n    alert(\"Wrong password\"); return;\n  }\n\n  /* --- decode only the solution that belongs to THIS button --- */\n  const wrapper = btn.parentElement;             // .b64-wrapper\n  wrapper.querySelectorAll('.hidden-solution').forEach(div => {\n    div.innerHTML = b64DecodeUtf8(div.dataset.encoded);\n    div.classList.remove('hidden-solution');\n    div.style.display = 'block';\n  });\n\n  /* Remove the button so the user can’t click it again */\n  btn.remove();\n}\n</script>\n\n<noscript>\n<div style=\"border: 1px solid #ccc; padding: 1em; margin-top: 1em; background: #f9f9f9;\">\n    <strong>JavaScript is required to unlock solutions.</strong><br>\n    Please enable JavaScript and reload the page,<br>\n    or download the source files from\n    <a href=\"https://github.com/mlr-org/mlr3website/\" target=\"_blank\" rel=\"noopener noreferrer\">GitHub</a>\n    and run the code locally.\n  </div>\n</noscript>\n```\n\n\n\n\n\n\n# Goal\n\nOur goal for this exercise sheet is to learn the basics of model calibration for supervised classification with `mlr3calibration`.\nIn a calibrated model, the predicted probability for an input feature vector can be interpreted as the true likelihood of the outcome belonging to the positive class, meaning that among all instances assigned a probability of $p$, approximately $p\\%$ will belong to the positive class.\n\n# Required packages\n\nWe will use `mlr3` for machine learning, and `mlr3calibration` specifically for calibration:\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nif (!require(\"mlr3calibration\")) {\n  remotes::install_github(\"AdriGl117/mlr3calibration\")\n}\nlibrary(mlr3calibration)\nlibrary(mlr3verse)\n\nset.seed(12345)\n```\n:::\n\n\n\n# Data: predicting cell segmentation quality\n\nThe `modeldata` package contains a data set called `cells`. Initially distributed by [Hill and Haney (2007)](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-8-340), they showed how to create models that predict the _quality_ of the image analysis of cells. The outcome has two levels: `\"PS\"` (for poorly segmented images) or `\"WS\"` (well-segmented). There are 56 image features that can be used to build a classifier. \n\nLet's load the data and remove an unwanted column: \n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nlibrary(modeldata)\ndata(cells, package = \"modeldata\")\ncells$case <- NULL\n```\n:::\n\n\n\n# 1 Calibrate a model with Platt scaling\n\nWe will apply Platt scaling to calibrate a model trained on the `cells` data. Platt scaling is a post-processing calibration method that fits a logistic regression model to the outputs of an uncalibrated classifier, transforming raw scores into calibrated probabilities.\n\n## 1.1 Creating a train-test split and tasks\n\nFirst, define a `task` object for the `cells` data set. Then, create a simple train-test split on the task to reserve test data for performance evaluation later on. As result, there should be a `task_train` and a `task_test`.\n\n<details>\n<summary>**Hint 1:**</summary>\n\nYou can use `partition()` on a given task object to create simple train-test split.\n\n</details>\n\n:::{.callout-note collapse=\"true\"}\n\n### Solution\n\n:::{.b64-solution}\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\ntask = as_task_classif(cells, target = \"class\")\nsplits = partition(task)\ntask_train = task$clone()$filter(splits$train)\ntask_test = task$clone()$filter(splits$test)\n```\n:::\n\n\n\n:::\n\n:::\n\n## 1.2 Assess model calibration\n\nTrain an XBOOST model on the training data. To do so, initialize an XGBOOST learner with `predict_type = \"prob\"`. Then, set `learner$id <- \"Uncalibrated Learner\"` for later reference. Train the learner on the correct task. Then, assess if the model is calibrated with `calibrationplot()`. The calibration plot shows the relationship between the predicted probabilities and the true outcomes. The plot is divided into bins, and within each bin, the mean predicted probability and the mean observed outcome are calculated. The calibration plot can be smoothed by setting `smooth = TRUE`.\n\n<details>\n<summary>**Hint 1:**</summary>\n\n`calibrationplot()` requires a `list` of learners even if the list contains only one argument.\n\n</details>\n\n:::{.callout-note collapse=\"true\"}\n\n### Solution\n\n:::{.b64-solution}\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\n#\nlearner = lrn(\"classif.xgboost\", predict_type = \"prob\")\nlearner$id <- \"Uncalibrated Learner\"\nlearner$train(task_train)\ncalibrationplot(list(learner), task_test, smooth = TRUE)\n```\n\n::: {.cell-output-display}\n![](index_files/figure-html/unnamed-chunk-7-1.png){fig-align='center' width=672}\n:::\n:::\n\n\n\nThe model is not well-calibrated. For predicted probabilities from 0 to 0.4, it seems to underestimate the true probability, for predicted probabilities higher than 0.4, it tends to overestimate the true probability.\n\n:::\n\n:::\n\n## 1.3 Calibration strategy\n\nIn `mlr3calibration`, to calibrate a learner you need a base learner (which will fit a model that is calibrated afterwards), a resampling strategy, and a calibration method (Platt, Beta or Isotonic). Initialize 1) another XGBOOST base learner, 2) a 5-fold CV resampling object, and 3) a calibration strategy. The calibration strategy in `mlr3calibration` is implemented as `PipeOpCalibration` object. It requires the base learner (`learner`), the calibration method (`method`), and the resampling method (`rsmp`) as arguments to be initialized. Practically, we want to use the calibration strategy as learner, so we have to express the pipeline operator within `as_learner()`. After that, set `learner_cal$id <- \"Platt Calibrated Learner\"` for later reference.\n\n<details>\n<summary>**Hint 1:**</summary>\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nlearner_uncal = ...\nrsmp = ...\nlearner_cal = as_learner(PipeOpCalibration$new(...))\nlearner_cal$id <- \"Platt Calibrated Learner\"\n```\n:::\n\n\n\n</details>\n\n<details>\n<summary>**Hint 2:**</summary>\n\nCheck the documentation of `PipeOpCalibration` with `??PipeOpCalibration`.\n\n</details>\n\n:::{.callout-note collapse=\"true\"}\n\n### Solution\n\n:::{.b64-solution}\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nlearner_uncal = lrn(\"classif.xgboost\", predict_type = \"prob\")\nrsmp = rsmp(\"cv\", folds = 5)\nlearner_cal = as_learner(PipeOpCalibration$new(learner = learner_uncal, method = \"platt\", rsmp = rsmp))\nlearner_cal$id <- \"Platt Calibrated Learner\"\n```\n:::\n\n\n\n:::\n\n:::\n\n## 1.4 Calibrate learner\n\nThe calibrated learner can be trained on a `task` as any other learner. Train the learner on `task_train`. Afterwards, plot the calibration plot again, comparing the uncalibrated XGBOOST model with the Platt-scaled XGBOOST model.\n\n:::{.callout-note collapse=\"true\"}\n\n### Solution\n\n:::{.b64-solution}\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nlearner_cal$train(task_train)\ncalibrationplot(list(learner, learner_cal), task_test, smooth = TRUE)\n```\n\n::: {.cell-output-display}\n![](index_files/figure-html/unnamed-chunk-10-1.png){fig-align='center' width=672}\n:::\n:::\n\n\n\nWhile the calibrated learner does not exhibit perfect calibration, it is much better calibrated then the uncalibrated learner, especially for predicted probabilities smaller than 0.5.\n\n:::\n\n:::\n\n# 2 Calibration measures\n\n`mlr3calibration` features measures for performance evaluation specifically to assess model calibration: the Expected Calibration Error (ECE) and the Integrated Calibration Index (ICI). The ECE is a measure of the difference between the predicted probabilities and the true outcomes. The ICI is a weighted average of the absolute differences between the calibration curve and\nthe diagonal perfectly calibrated line. Compute the ECE for both models. The calibration measures are implemented similarly to other measures in `mlr3`. Therefore, you need to 1) predict on the test data and then 2) score the predictions while specifying the correct calibration measure.\n\n<details>\n<summary>**Hint 1:**</summary>\n\nCheck `??mlr3calibration::ece` on how to initialize the ECE measure within `$score()`.\n\n</details>\n\n:::{.callout-note collapse=\"true\"}\n\n### Solution\n\n:::{.b64-solution}\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\n# Predictions\npreds_uncal = learner$predict(task_test)\npreds_cal = learner_cal$predict(task_test)\n# Calculate the ECE\nece_uncal = preds_uncal$score(ece$new())\nece_cal = preds_cal$score(ece$new())\n# Uncalibrated ECE\nece_uncal\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nclassif.ece \n  0.1184465 \n```\n\n\n:::\n\n```{.r .cell-code}\n# Calibrated ECE\nece_cal\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nclassif.ece \n 0.08859307 \n```\n\n\n:::\n:::\n\n\n\nThe calibrated model has a much lower ECE. Therefore, we can infer that Platt scaling was successful in producing a more well-calibrated model.\n\n:::\n\n:::\n\n# 3 Tuning and Pipelines\n\n`PipeOpCalibration` can be treated as any other `PipeOp` object. Therefore, we can use them within more complex tuning and pipeline constructs. There are many sensible options. For example, we could pass a tuned base learner to the calibrator or tune the base learner within the calibrator. Similarly, we can include a calibrated learner in a pipeline or choose to calibrate the entire pipeline. Let's try how to connect a feature filter to a calibrator. Construct a pipeline that 1) filters the 10 most relevant features according to their information gain, 2) then fits a random forest, and 3) calibrate this pipeline with beta calibration using 5-fold CV. Express this calibrated pipeline as learner, train it on the training task and plot the calibration plot with the Platt scaled and beta-calibrated models.\n\n\n\n<details>\n<summary>**Hint 1:**</summary>\n\nYou may use this skeleton code for the required steps.\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\npo_filter = po(...)\npipeline = as_learner(... %>>% ...)\npipeline_cal = as_learner(PipeOpCalibration$new(...))\npipeline_cal$id <- \"Beta Calibrated Learner\"\npipeline_cal$train(...)\ncalibrationplot(...)\n```\n:::\n\n\n\n</details>\n\n:::{.callout-note collapse=\"true\"}\n\n### Solution\n\n:::{.b64-solution}\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\npo_filter = po(\"filter\", filter = flt(\"information_gain\"), param_vals = list(filter.nfeat = 10L))\npipeline = as_learner(po_filter %>>% lrn(\"classif.ranger\", predict_type = \"prob\"))\npipeline_cal = as_learner(PipeOpCalibration$new(learner = pipeline,\n                                                rsmp = rsmp(\"cv\", folds = 10),\n                                                method = \"beta\"))\npipeline_cal$id <- \"Beta Calibrated Learner\"\npipeline_cal$train(task_train)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] -13.36795\n[1] 64.20859\n[1] -45.74624\n[1] 49.62498\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] -41.99003\n[1] 32.05335\n[1] 1.802867\n[1] 85.86877\n[1] -70.70553\n[1] 0.9673037\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] -39.66383\n[1] 48.62042\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] -44.09992\n[1] 32.06818\n```\n\n\n:::\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] -10.50196\n[1] 83.03415\n[1] -34.26856\n[1] 29.53432\n[1] -60.5959\n[1] 12.44805\n```\n\n\n:::\n\n```{.r .cell-code}\ncalibrationplot(list(learner_cal, pipeline_cal), task_test, smooth = TRUE)\n```\n\n::: {.cell-output-display}\n![](index_files/figure-html/unnamed-chunk-13-1.png){fig-align='center' width=672}\n:::\n:::\n\n\n\nThe beta-calibrated random forest model seems very well-calibrated.\n\n:::\n\n:::\n",
+    "supporting": [
+      "index_files"
+    ],
+    "filters": [
+      "rmarkdown/pagebreak.lua"
+    ],
+    "includes": {},
+    "engineDependencies": {},
+    "preserve": {},
+    "postProcess": true
+  }
+}

mlr-org/gallery-all-nofilter.ejs

@@ -8,11 +8,16 @@
   }
 
   // Define your custom group order
-  const customGroupOrder = ["Introduction", "Tuning", "Feature Preprocessing", "Advanced Feature Preprocessing", "Performance Evaluation", "Advanced Performance Evaluation", "Ensembles Stacking", "Parallelization", "Imputation"];
+  const customGroupOrder = ["Introduction", "Tuning", "Feature Preprocessing", "Advanced Feature Preprocessing", "Performance Evaluation", "Advanced Performance Evaluation", "Ensembles Stacking", "Parallelization", "Imputation", "Imbalanced"];
 
   // Optionally include any other groups not listed in the custom order
   const extraGroups = Object.keys(grouped).filter(g => !customGroupOrder.includes(g)).sort();
   const finalGroupOrder = [...customGroupOrder, ...extraGroups];
+
+  // Sort items within each group by date descending
+  for (const group of Object.keys(grouped)) {
+    grouped[group].sort((a, b) => new Date(a.date) - new Date(b.date));
+  }
 %>
 
 <% for (const group of finalGroupOrder) { %>

mlr-org/gallery/appliedml/2025-04-10-intro-benchmarking-sol/index.qmd

@@ -9,7 +9,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Hyperparameter tuning and benchmarking on german credit task.
-date: ""
+date: 04-25-2025
 params:
   showsolution: true
   base64encode: true

mlr-org/gallery/appliedml/2025-04-10-intro-resampling-sol/index.qmd

@@ -9,7 +9,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Use 5-fold cross validation to evaluate logistic regression and knn learner on german credit set.
-date: ""
+date: 04-24-2025
 params:
   showsolution: true
   base64encode: true

mlr-org/gallery/appliedml/2025-04-10-intro-train-predict-evaluate-sol/index.qmd

@@ -10,7 +10,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Introduction to German Credit dataset and classification. Train predict and evaluate a logistic regression learner with hold-out split.
-date: ""
+date: 04-23-2025
 params:
   showsolution: true
   base64encode: true

mlr-org/gallery/appliedml/2025-04-10-intro-tree-methods-sol/index.qmd

@@ -9,7 +9,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Use, plot and benchmark classification tree and random forest on german credit set.
-date: ""
+date: 04-30-2025
 params:
   showsolution: true
   base64encode: true

mlr-org/gallery/appliedml/2025-04-28-feature-preproc-encoding-scaling/index.qmd

@@ -11,7 +11,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Create a pipeline to do feature preprocessing (one-hot-encoding, Yeo-Johnson transformation) for the german credit task.
-date: ""
+date: 05-21-2025
 params:
   showsolution: true
   base64encode: true

mlr-org/gallery/appliedml/2025-04-28-feature-preproc-impact-encoding/index.qmd

@@ -10,7 +10,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Construct pipelines for benchmark experiments on kc_housing set.
-date: ""
+date: 05-22-2025
 params:
   showsolution: true
   base64encode: true

mlr-org/gallery/appliedml/2025-04-28-feature-preproc-xgboost/index.qmd

@@ -10,7 +10,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Optimize hyperparameters of xgboost for german credit task.
-date: ""
+date: 05-23-2025
 params:
   showsolution: true
   base64encode: true

mlr-org/gallery/appliedml/2025-04-28-tuning-adv-resampling-custom-msr-sol/index.qmd

@@ -11,7 +11,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Use stratified resampling to evaluate the german credit set and blocking for BreastCancer set. Define custom measures in mlr3 and use them to evaluate a model on the mtcars task.
-date: ""
+date: 05-07-2025
 params:
   showsolution: true
   base64encode: true

mlr-org/gallery/appliedml/2025-04-28-tuning-mbo/index.qmd

@@ -10,7 +10,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Use Bayesian optimization (BO) using `bbotk` and `mlr3mbo` for general black box optimization problems, and more specifically, hyperparameter optimization (HPO).
-date: ""
+date: 05-16-2025
 params:
   showsolution: true
   base64encode: true

mlr-org/gallery/appliedml/2025-04-28-tuning-nested-resampling/index.qmd

@@ -10,7 +10,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Estimate the generalization error of a k-NN model on german credit set via nested resampling.
-date: ""
+date: 05-15-2025
 params:
   showsolution: true
   base64encode: true

mlr-org/gallery/appliedml/2025-04-28-tuning-tuning/index.qmd

@@ -9,7 +9,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Optimize hyperparameters for k-NN and SVM classifier on german credit set.
-date: ""
+date: 05-14-2025
 params:
   showsolution: true
   base64encode: true

mlr-org/gallery/appliedml/2025-05-05-adv-feature-preproc-feature-selection/index.qmd

@@ -10,7 +10,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Select features from the german credit set and evaluate model performance.
-date: ""
+date: 05-29-2025
 params:
   showsolution: true
   base64encode: true

mlr-org/gallery/appliedml/2025-05-05-adv-feature-preproc-filter/index.qmd

@@ -9,7 +9,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Use filters in a mlr3 pipeline
-date: ""
+date: 05-28-2025
 params:
   showsolution: true
   base64encode: true

mlr-org/gallery/appliedml/2025-05-05-adv-feature-preproc-hw-filters/index.qmd

@@ -10,7 +10,7 @@ author:
     url: https://www.essentialds.de
 description: |
   Use pipelines for efficient pre-processing and model training on a the kc_housing task.
-date: ""
+date: 05-30-2025
 params:
   showsolution: true
   base64encode: true