From afbffcecbf5497c333943610e674a6fcfbd7745b Mon Sep 17 00:00:00 2001 From: "Ashwin V. Mohanan" Date: Tue, 10 Mar 2026 15:41:27 +0100 Subject: [PATCH] Checkout changes with Pytorch Implemented in @mimer-ai/deep-learning-intro by the following authors: - @ashwinvis Ashwin - @otaub Oskar - @ffrancesco94 Francesco - @marlon-tobaben Marlon - @lodo1995 Lodovico Initial version. More changes required as noted in #631 --- episodes/1-introduction.md | 21 +- episodes/2-keras.md | 643 ++++++++++++-- episodes/3-monitor-the-model.md | 813 +++++++++++++++++- episodes/4-advanced-layer-types.md | 576 ++++++++++++- episodes/5-transfer-learning.md | 466 +++++++++- episodes/6-outlook.md | 2 +- ...ning_history_transfer_learning_pytorch.png | Bin 0 -> 26609 bytes learners/setup.md | 102 ++- profiles/learner-profiles.md | 8 +- 9 files changed, 2442 insertions(+), 189 deletions(-) create mode 100644 episodes/fig/05_training_history_transfer_learning_pytorch.png diff --git a/episodes/1-introduction.md b/episodes/1-introduction.md index ebc324a65..061379041 100644 --- a/episodes/1-introduction.md +++ b/episodes/1-introduction.md @@ -110,8 +110,8 @@ Combine the following statements to the correct activation function: 6. (optional) This function is not differentiable at 0 7. (optional) This function is the default for Dense layers (search the Keras documentation!) -*Activation function plots by Laughsinthestocks - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=44920411, -https://commons.wikimedia.org/w/index.php?curid=44920600, https://commons.wikimedia.org/w/index.php?curid=44920533* +_Activation function plots by Laughsinthestocks - Own work, CC BY-SA 4.0, , +, _ ::: solution ## Solution @@ -134,7 +134,7 @@ The image below shows an example of a neural network with three layers, each cir ![ Image credit: Glosser.ca, CC BY-SA 3.0 , via Wikimedia Commons, -[original source](https://commons.wikimedia.org/wiki/File:Colored_neural_network.svg) +original source: ](fig/01_neural_net.png){ alt='A diagram of a three layer neural network with an input layer, one hidden layer, and an output layer.' } @@ -198,6 +198,7 @@ a. | 0 | 1 | **1** | | 1 | 1 | **0** | | 1 | 0 | **1** | + b. This solves the XOR logical problem, the output is 1 if only one of the two inputs is 1. ::: @@ -229,7 +230,7 @@ Note that the color coding refers to different layer types that will be introduc as we proceed in this lesson.](fig/01_deep_network.png){alt='An example of a deep neural network'} ![As a result of the optimization process, the different layers of a neural network tend to learn increasingly abstract representations of the input data. -](fig/01_nn_abstraction_layers.png) +](fig/01_nn_abstraction_layers.png){alt='Example of two different neural networks and how each layer process different abstract representations of input data'} ### How do neural networks learn? What happens in a neural network during the training process? @@ -249,7 +250,7 @@ To solve it: 3. Click on "Math Renderer". 4. Click on "Common HTML". -from: https://physics.meta.stackexchange.com/questions/14408/bug-in-mathjax-rendering-using-chrome +from: ::: :::: challenge @@ -455,10 +456,10 @@ Think about a problem you would like to use deep learning to solve. 2. What data inputs and outputs will you have? 3. Do you think you will need to train the network or will a pre-trained network be suitable? 4. What data do you have to train with? What preparation will your data need? Consider both the data you are going to predict/classify from and the data you will use to train the network. +::: ::: solution Discuss your answers with the group or the person next to you. -::: :::: @@ -470,11 +471,11 @@ There are many software libraries available for deep learning including: [TensorFlow](https://www.tensorflow.org/) was developed by Google and is one of the older deep learning libraries, ported across many languages since it was first released to the public in 2015. It is very versatile and capable of much more than deep learning but as a result it often takes a lot more lines of code to write deep learning operations in TensorFlow than in other libraries. It offers (almost) seamless integration with GPU accelerators and Google's own TPU (Tensor Processing Unit) chips that are built specially for machine learning. -### PyTorch +### PyTorch [PyTorch](https://pytorch.org/) was developed by Facebook in 2016 and is a popular choice for deep learning applications. It was developed for Python from the start and feels a lot more "pythonic" than TensorFlow. Like TensorFlow it was designed to do more than just deep learning and offers some very low level interfaces. [PyTorch Lightning](https://www.pytorchlightning.ai/) offers a higher level interface to PyTorch to set up experiments. Like TensorFlow it is also very easy to integrate PyTorch with a GPU. In many benchmarks it outperforms the other libraries. -### Keras +### Keras [Keras](https://keras.io/) is designed to be easy to use and usually requires fewer lines of code than other libraries. We have chosen it for this lesson for that reason. Keras can actually work on top of TensorFlow (and several other libraries), hiding away the complexities of TensorFlow while still allowing you to make use of their features. @@ -484,10 +485,10 @@ Keras also benefits from a very good set of [online documentation](https://keras ### Installing Keras and other dependencies -Follow the [setup instructions](learners/setup.md#packages) to install Keras, Seaborn and scikit-learn. +Follow the [setup instructions](./setup.md#packages) to install Keras, Seaborn and scikit-learn. ## Testing Keras Installation -Keras is available as a module within TensorFlow, as described in the [setup instructions](learners/setup.md#packages). +Keras is available as a module within TensorFlow, as described in the [setup instructions](./setup.md#packages). Let's therefore check whether you have a suitable version of TensorFlow installed. Open up a new Jupyter notebook or interactive python console and run the following commands: ```python diff --git a/episodes/2-keras.md b/episodes/2-keras.md index 7e94f00b9..d4a2aa79c 100644 --- a/episodes/2-keras.md +++ b/episodes/2-keras.md @@ -1,31 +1,33 @@ --- -title: "Classification by a neural network using Keras" +title: "Classification by a neural network using Pytorch/Keras" teaching: 60 exercises: 50 --- ::: questions -- How do I compose a neural network using Keras? + +- How do I compose a neural network using Pytorch/Keras? - How do I train this network on a dataset? - How do I get insight into learning process? - How do I measure the performance of the network? ::: ::: objectives + - Use the deep learning workflow to structure the notebook - Explore the dataset using pandas and seaborn - Identify the inputs and outputs of a deep neural network. -- Use one-hot encoding to prepare data for classification in Keras +- Use one-hot encoding to prepare data for classification in Pytorch/Keras - Describe a fully connected layer -- Implement a fully connected layer with Keras -- Use Keras to train a small fully connected network on prepared data +- Implement a fully connected layer with Pytorch/Keras +- Use Pytorch/Keras to train a small fully connected network on prepared data - Interpret the loss curve of the training process - Use a confusion matrix to measure the trained networks' performance on a test set ::: - ## Introduction -In this episode we will learn how to create and train a neural network using Keras to solve a simple classification task. + +In this episode we will learn how to create and train a neural network using PyTorch or Keras to solve a simple classification task. The goal of this episode is to quickly get your hands dirty in actually defining and training a neural network, without going into depth of how neural networks work on a technical or mathematical level. @@ -35,7 +37,7 @@ In fact, this is also what we would recommend you to do when working on real-wor First quickly build a working pipeline, while taking shortcuts. Then, slowly make the pipeline more advanced while you keep on evaluating the approach. -In [episode 3](episodes/3-monitor-the-model.md) we will expand on the concepts that are lightly introduced in this episode. +In [episode 3](./3-monitor-the-model.md) we will expand on the concepts that are lightly introduced in this episode. Some of these concepts include: how to monitor the training progress and how optimization works. ::: instructor @@ -60,7 +62,9 @@ As a reminder below are the steps of the deep learning workflow: In this episode we will focus on a minimal example for each of these steps, later episodes will build on this knowledge to go into greater depth for some or all of these steps. ::: callout + ## GPU usage + For this lesson having a [GPU (graphics processing unit)](https://glosario.carpentries.org/en/#gpu) available is not needed. We specifically use very small toy problems so that you do not need one. However, Keras will use your GPU automatically when it is available. @@ -69,13 +73,16 @@ require a more complex neural network. ::: ## 1. Formulate/outline the problem: penguin classification + In this episode we will be using the [penguin dataset](https://zenodo.org/record/3960218). This is a dataset that was published in 2020 by Allison Horst and contains data on three different species of the penguins. We will use the penguin dataset to train a neural network which can classify which species a penguin belongs to, based on their physical characteristics. ::: callout + ## Goal + The goal is to predict a penguins' species using the attributes available in this dataset. ::: @@ -84,13 +91,12 @@ The physical attributes measured are flipper length, beak length, beak width, bo ![*Artwork by @allison_horst*][palmer-penguins] - ![*Artwork by @allison_horst*][penguin-beaks] - These data were collected from 2007 - 2009 by Dr. Kristen Gorman with the [Palmer Station Long Term Ecological Research Program](https://lternet.edu/site/palmer-antarctica-lter/), part of the [US Long Term Ecological Research Network](https://lternet.edu/). The data were imported directly from the [Environmental Data Initiative](https://edirepository.org/) (EDI) Data Portal, and are available for use by CC0 license ("No Rights Reserved") in accordance with the [Palmer Station Data Policy](https://lternet.edu/data-access-policy/). ## 2. Identify inputs and outputs + To identify the inputs and outputs that we will use to design the neural network we need to familiarize ourselves with the dataset. This step is sometimes also called data exploration. @@ -103,6 +109,7 @@ import seaborn as sns ``` We can load the penguin dataset using + ```python penguins = sns.load_dataset('penguins') ``` @@ -110,7 +117,9 @@ penguins = sns.load_dataset('penguins') This will give you a pandas dataframe which contains the penguin data. ### Inspecting the data + Using the pandas `head` function gives us a quick look at the data: + ```python penguins.head() ``` @@ -134,10 +143,12 @@ Let's look at the shape of the dataset: There are 344 samples and 7 columns (plus the index column), so 6 features. ### Visualization + Looking at numbers like this usually does not give a very good intuition about the data we are working with, so let us create a visualization. #### Pair Plot + One nice visualization for datasets with relatively few attributes is the Pair Plot. This can be created using `sns.pairplot(...)`. It shows a scatterplot of each attribute plotted against each of the other attributes. By using the `hue='species'` setting for the pairplot the graphs on the diagonal are layered kernel density estimate plots for the different values of the `species` column. @@ -154,15 +165,17 @@ sns.pairplot(penguins, hue="species") Take a look at the pairplot we created. Consider the following questions: -* Is there any class that is easily distinguishable from the others? -* Which combination of attributes shows the best separation for all 3 class labels at once? -* (optional) Create a similar pairplot, but with `hue="sex"`. Explain the patterns you see. +- Is there any class that is easily distinguishable from the others? +- Which combination of attributes shows the best separation for all 3 class labels at once? +- (optional) Create a similar pairplot, but with `hue="sex"`. Explain the patterns you see. Which combination of features distinguishes the two sexes best? ::: solution + ## Solution -* The plots show that the green class, Gentoo is somewhat more easily distinguishable from the other two. -* The other two seem to be separable by a combination of bill length and bill + +- The plots show that the green class, Gentoo is somewhat more easily distinguishable from the other two. +- The other two seem to be separable by a combination of bill length and bill depth (other combinations are also possible such as bill length and flipper length). Answer to optional question: @@ -181,6 +194,7 @@ The combination of `bill_depth_mm` and `body_mass_g` gives the best separation. :::: ### Input and Output Selection + Now that we have familiarized ourselves with the dataset we can select the data attributes to use as input for the neural network and the target that we want to predict. @@ -188,38 +202,45 @@ In the rest of this episode we will use the `bill_length_mm`, `bill_depth_mm`, ` The target for the classification task will be the `species`. ::: callout + ## Data Exploration + Exploring the data is an important step to familiarize yourself with the problem and to help you determine the relevant inputs and outputs. ::: ## 3. Prepare data -The input data and target data are not yet in a format that is suitable to use for training a neural network. +The input data and target data are not yet in a format that is suitable to use for training a neural network. For now we will only use the numerical features `bill_length_mm`, `bill_depth_mm`, `flipper_length_mm`, `body_mass_g` only, so let's drop the categorical columns: + ```python # Drop categorical columns penguins_filtered = penguins.drop(columns=['island', 'sex']) ``` ### Clean missing values + During the exploration phase you may have noticed that some rows in the dataset have missing (NaN) values, leaving such values in the input data will ruin the training, so we need to deal with them. There are many ways to deal with missing values, but for now we will just remove the offending rows by adding a call to `dropna()`: + ```python # Drop the rows that have NaN values in them penguins_filtered = penguins_filtered.dropna() ``` Finally, we select only the features + ```python # Extract columns corresponding to features features = penguins_filtered.drop(columns=['species']) ``` ### Prepare target data for training + Second, the target data is also in a format that cannot be used in training. A neural network can only take numerical inputs and outputs, and learns by calculating how "far away" the species predicted by the neural network is @@ -233,6 +254,7 @@ the other columns. For instance, for a penguin of the Adelie species the one-hot encoding would be 1 0 0. Fortunately, Pandas is able to generate this encoding for us. + ```python import pandas as pd @@ -241,29 +263,34 @@ target.head() # print out the top 5 to see what it looks like. ``` :::: challenge + ## One-hot encoding + How many output neurons will our network have now that we one-hot encoded the target class? -* A: 1 -* B: 2 -* C: 3 +- A: 1 +- B: 2 +- C: 3 +::: ::: solution + ## Solution + C: 3, one for each output variable class -::: :::: ### Split data into training and test set -Finally, we will split the dataset into a training set and a test set. -As the names imply we will use the training set to train the neural network, -while the test set is kept separate. -We will use the test set to assess the performance of the trained neural network -on unseen samples. -In many cases a validation set is also kept separate from the training and test sets (i.e. the dataset is split into 3 parts). -This validation set is then used to select the values of the parameters of the neural network and the training methods. -For this episode we will keep it at just a training and test set however. + +Then, we will split the dataset into a training set and a test set. As the +names imply we will use the training set to train the neural network, while the +test set is kept separate. We will use the test set to assess the performance +of the trained neural network on unseen samples. In many cases a validation set +is also kept separate from the training and test sets (i.e. the dataset is +split into 3 parts). This validation set is then used to select the values of +the parameters of the neural network and the training methods. For this episode +we will keep it at just a training and test set. To split the cleaned dataset into a training and test set we will use a very convenient function from sklearn called `train_test_split`. @@ -285,21 +312,54 @@ X_train, X_test, y_train, y_test = train_test_split(features, target, test_size= ``` ::: callout + ## Importance of using the same train-test split + By setting `random_state=0` we ensure that everyone has the same train-test split. When doing machine learning and deep learning it is crucial that you use the same train and test dataset for different experiments. Comparing evaluation metrics between experiments run on different data splits is meaningless, because the accuracy of a model depends on the data used to train and test it. ::: +### Scale the input features + +If you take a look back at the initial data inspection, you can see that the +various features have different scales. `bill_length_mm` and `bill_depth_mm` +are in the order of {math}`10^1`, while `flipper_length_mm` and `body_mass_g` are in +the order of {math}`10^2` and {math}`10^3` respectively. Machine learning models work best +when all features present similar scales, with values centered around zero. + +Therefore, it is good practice to scale the input features to bring all of them +to a similar range. `scikit-learn` offers several convienent scaler classes to +do this. In this case, we use `RobustScaler`, which provides a good default +that is suitable for a variety of datasets. In particular, as the name +suggests, it is robust to "outliers", i.e. entries in the dataset that present +abnormal values. The `scikit-learn` documentation provides a detailed +comparison of the [effects of different scalers on data with +outliers](https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html). + +```python +from sklearn.preprocessing import RobustScaler + +feature_scaler = RobustScaler() +X_train_scaled = feature_scaler.fit_transform(X_train) +X_test_scaled = feature_scaler.transform(X_test) +``` + ::: instructor + ## BREAK + This is a good time for switching instructor and/or a break. ::: ## 4. Build an architecture from scratch -### Keras for neural networks +### Import the deep learning framework + +::::::: group-tab + +###### Keras Keras is a machine learning framework with ease of use as one of its main features. It is part of the tensorflow python package and can be imported using `from tensorflow import keras`. @@ -308,19 +368,64 @@ Keras includes functions, classes and definitions to define deep learning models Before we move on to the next section of the workflow we need to make sure we have Keras imported. We do this as follows: + ```python from tensorflow import keras ``` + + +###### PyTorch + +PyTorch is a popular deep learning framework designed to enable developers to implement any type of neural network in environments ranging from academic research to industrial applications. +Thus, PyTorch includes functions and classes to define deep learning models, cost functions and optimizers (optimizers are used to train a model). + +Before we move on to the next section of the workflow we need to make sure we have PyTorch imported. +We do this as follows: + +```python +import torch +``` + + +::::::: + +### Set the seeds + For this episode it is useful if everyone gets the same results from their training. -Keras uses a random number generator at certain points during its execution. -Therefore we will need to set two random seeds, one for numpy and one for tensorflow: +Keras and PyTorch uses a random number generator at certain points during its execution. + +::::::: group-tab + +###### Keras + ```python +from numpy.random import seed +seed(1) + keras.utils.set_random_seed(2) ``` + + +###### PyTorch + +```python +from numpy.random import seed +seed(1) + +torch.manual_seed(2) +``` + + +::::::: + ::: callout + +(when-to-use-random-seeds)= + ## When to use random seeds? + We use a random seed here to ensure that we get the same results every time we run this code. This makes our results reproducible and allows us to better compare results between different experiments. @@ -331,7 +436,11 @@ So, to get truly replicable deep learning pipelines you need to run the notebook ### Build a neural network from scratch -We will now build a simple neural network from scratch using Keras. +We will now build a simple neural network from scratch. + +::::::: group-tab + +###### Keras With Keras you compose a neural network by creating layers and linking them together. For now we will only use one type of layer called a fully connected @@ -354,6 +463,7 @@ inputs = keras.Input(shape=(X_train.shape[1],)) We store a reference to this input class in a variable so we can pass it to the creation of our hidden layer. Creating the hidden layer can then be done as follows: + ```python hidden_layer = keras.layers.Dense(10, activation="relu")(inputs) ``` @@ -377,6 +487,7 @@ Finally we store a reference in the `hidden_layer` variable so we can pass it to Now we create another layer that will be our output layer. Again we use a Dense layer and so the call is very similar to the previous one. + ```python output_layer = keras.layers.Dense(3, activation="softmax")(hidden_layer) ``` @@ -390,34 +501,89 @@ species. Now that we have defined the layers of our neural network we can combine them into a Keras model which facilitates training the network. + ```python model = keras.Model(inputs=inputs, outputs=output_layer) +``` + +Now that the neural network is created, we can inspect it: + +```python model.summary() ``` + + +###### PyTorch + +In Pytorch, the architecture of a neural network is defined in a class that +inherits from `torch.nn.Module`. The network itself is created by stacking +layers and linking them together. In this episode, we will only use one type of +layer called *fully connected* or *dense*, which PyTorch dubs `Linear`; the +number of neurons is prescribed by the user. For fully connected layers, each +neuron gets an edge (i.e. connection) to **all** of the input neurons and +**all** of the output neurons. The hidden layer in the image in the +introduction of this episode is a fully connected layer. A possible +architecture for a penguin classifier using one hidden layer is proposed below: + +```python +class PenguinModel(torch.nn.Module): + def __init__(self, input_shape): + super().__init__() + self.hidden_layer = torch.nn.Linear(input_shape, 10) + self.output_layer = torch.nn.Linear(10, 3) + + def forward(self, x): + x = self.hidden_layer(x) + x = torch.nn.functional.relu(x) + x = self.output_layer(x) + x = torch.nn.functional.softmax(x, dim=1) + return x + +model = PenguinModel(X_train.shape[1]).to(device) +``` + +In Pytorch, the layers are defined in the constructor of the class. The dimension of the input is defined implicitly by the size of the hidden layer (`torch.nn.Linear(X_train.shape[1], 10)`). The number of neurons is prescribed in the second parameter of the linear layer (10); this quantity is a hyperparameter that we have to choose and tune based on the specific task the network has to perform. We will get back to this in the section on refining the model. The output layer is then constructed based on the size of the hidden layer (10) and the number of classes (3), since we are using one-hot encoding. + +What happens to the input throughout the network is defined in the `forward()` method: it goes through the first layer, which is then activated by a `ReLU`, which is commonly used in deep neural networks. After that we have the output layer with three neurons (since we have three classes). This layer uses a `softmax()` activation, which makes sure that the three output neurons produce values in the range (0,1) and that their sum is 1. These values can be interpreted as the `probability` that the sample belongs to a certain class. + +Now that the neural network is created, we can inspect it: + +```python +from torchinfo import summary +summary(model, input_size=X_train.shape[1:], batch_dim=0, device=device) +``` + + + + +::::::: + The model summary here can show you some information about the neural network we have defined. ::: callout + ## Trainable and non-trainable parameters -Keras distinguishes between two types of weights, namely: + +Both Pytorch and Keras distinguish between two types of weights, namely: - trainable parameters: these are weights of the neurons that are modified when we train the model in order to minimize our loss function (we will learn about loss functions shortly!). -- non-trainable parameters: these are weights of the neurons that are not changed when we train the model. These could be for many reasons - using a pre-trained model, choice of a particular filter for a convolutional neural network, and statistical weights for batch normalization are some examples. +- non-trainable parameters: these are weights of the neurons that are not changed when we train the model. These could be for many reasons - using a pre-trained model, choice of a particular filter for a convolutional neural network, and statistical weights for batch normalization are some examples. If these reasons are not clear right away, don't worry! In later episodes of this course, we will touch upon a couple of these concepts. -::: - +::: ::: instructor For optional question 3 in the challenge below named 'Visualizing the model', the goal is to visualize the network. It supplements the textual explanation of output from `model.summary()`. You could choose to show and discuss the resulting visualization to the learners, so that learners who did not finish the optional exercise can also learn from the visualization of the model. ::: +::::::::: challenge -:::: challenge ## Create the neural network -With the code snippets above, we defined a Keras model with 1 hidden layer with + +With the code snippets above, we defined a model with 1 hidden layer with 10 neurons and an output layer with 3 neurons. 1. How many parameters does the resulting model have? @@ -425,7 +591,8 @@ With the code snippets above, we defined a Keras model with 1 hidden layer with in the hidden layer? #### (optional) Visualizing the model -Optionally, you can also visualize the same information as `model.summary()` in graph form. + +Optionally, you can also visualize the same information as `model.summary()` / `torchinfo.summary()` in graph form. This step requires the command-line tool `dot` from Graphviz installed, you installed it by following the setup instructions. You can check that the installation was successful by executing `dot -V` in the command line. You should get something as follows: @@ -434,6 +601,9 @@ as follows: $ dot -V dot - graphviz version 2.43.0 (0) ``` +:::: group-tab + +### Keras 3. (optional) Provided you have `dot` installed, execute the `plot_model` function as shown below. @@ -448,7 +618,8 @@ keras.utils.plot_model( ) ``` -#### (optional) Keras Sequential vs Functional API +**(optional) Keras Sequential vs Functional API** + So far we have used the [Functional API](https://keras.io/guides/functional_api/) of Keras. You can also implement neural networks using [the Sequential model](https://keras.io/guides/sequential_model/). As you can read in the documentation, the Sequential model is appropriate for **a plain stack of layers** @@ -456,9 +627,47 @@ where each layer has **exactly one input tensor and one output tensor**. 4. (optional) Use the Sequential model to implement the same network -::: solution + + + +### PyTorch + +3. (optional) Provided you have `dot` and `torchview` installed, execute `draw_graph` function + as shown below. + +```python +from torchview +model_graph = draw_graph(model, input_size=(1, 10), expand_nested=True) +model_graph.visual_graph +``` + +**(optional) PyTorch Sequential vs Object-oriented API** + +So far we have used the [Object-oriented API](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html) of PyTorch. +You can also implement neural networks using +[`torch.nn.Sequential`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Sequential.html). +As you can read in the documentation, the Sequential model is appropriate for **a plain stack of layers** +where each layer has **exactly one input tensor and one output tensor**. + +4. (optional) Use `torch.nn.Sequential` model to implement the same network + + + + +:::: + + + +:::::::: solution + ## Solution + +::::::: group-tab + +###### Keras + Have a look at the output of `model.summary()`: + ```python model.summary() ``` @@ -483,9 +692,10 @@ Model: "functional" Non-trainable params: 0 (0.00 B) ``` -The model has 83 trainable parameters. Each of the 10 neurons in the in the `dense` hidden layer is connected to each of -the 4 inputs in the input layer resulting in 40 weights that can be trained. The 10 neurons in the hidden layer are also -connected to each of the 3 outputs in the `dense_1` output layer, resulting in a further 30 weights that can be trained. + +The model has 83 trainable parameters. Each of the 10 neurons in the in the `dense` hidden layer is connected to each of +the 4 inputs in the input layer resulting in 40 weights that can be trained. The 10 neurons in the hidden layer are also +connected to each of the 3 outputs in the `dense_1` output layer, resulting in a further 30 weights that can be trained. By default `Dense` layers in Keras also contain 1 bias term for each neuron, resulting in a further 10 bias values for the hidden layer and 3 bias terms for the output layer. `40+30+10+3=83` trainable parameters. @@ -501,6 +711,7 @@ print(model.dtype) ```output float32 ``` + The model weights are represented using `float32` data type, which consumes 32 bits or 4 bytes for each weight. We have 83 parameters, and therefore in total, the model requires `83*4=332` bytes of memory to load into the computer's memory. @@ -513,14 +724,16 @@ So in total 8 extra parameters. *The name in quotes within the string `Model: "functional"` may be different in your view; this detail is not important.* -#### (optional) Visualizing the model +**(optional) Visualizing the model** + 3. Upon executing the `plot_model` function, you should see the following image. ![Output of *keras.utils.plot_model()* function][plot-model] +**(optional) Keras Sequential vs Functional API** -#### (optional) Keras Sequential vs Functional API 4. This implements the same model using the Sequential API: + ```python model = keras.Sequential( [ @@ -532,12 +745,85 @@ model = keras.Sequential( ``` We will use the Functional API for the remainder of this course, since it is more flexible and more explicit. -::: -:::: + + +###### PyTorch + +Have a look at the output of `torchinfo.summary()`: + +```python +from torchinfo import summary +summary(model, input_size=X_train.shape[1:], batch_dim=0, device=device) +``` + +```output +========================================================================================== +Layer (type:depth-idx) Output Shape Param # +========================================================================================== +PenguinModel [1, 3] -- +├─Linear: 1-1 [1, 10] 50 +├─Linear: 1-2 [1, 3] 33 +========================================================================================== +Total params: 83 +Trainable params: 83 +Non-trainable params: 0 +Total mult-adds (Units.MEGABYTES): 0.00 +========================================================================================== +Input size (MB): 0.00 +Forward/backward pass size (MB): 0.00 +Params size (MB): 0.00 +Estimated Total Size (MB): 0.00 +========================================================================================== +``` + +The model has 83 trainable parameters. Each of the 10 neurons in the in the `Linear: 1-1` hidden layer is connected to each of +the 4 inputs in the input layer resulting in 40 weights that can be trained. The 10 neurons in the hidden layer are also +connected to each of the 3 outputs in the `Linear: 1-2` output layer, resulting in a further 30 weights that can be trained. +By default `Linear` layers in PyTorch also contain 1 bias term for each neuron, resulting in a further 10 bias values for the +hidden layer and 3 bias terms for the output layer. `40+30+10+3=83` trainable parameters. + +Note that the output shape always contains `1` as the first entry of the tuple. This is the *flexible* dimension which is used by the model when processing several samples at the same time, what is usually called a `batch`. You will learn more about batching in lesson 3. + +**FIXME**: *Discuss memory footprint* + +If you increase the number of neurons in the hidden layer the number of +trainable parameters in both the hidden and output layer increases or +decreases in accordance with the number of neurons added. +Each extra neuron has 4 weights connected to the input layer, 1 bias term, and 3 weights connected to the output layer. +So in total 8 extra parameters. + +**(optional) Visualizing the model** + +3. **FIXME**: *To be completed using* `torchview.draw_graph`. + +**(optional) PyTorch Sequential vs Object-oriented API** + +4. This implements the same model using the Sequential API: + +```python +model = torch.nn.Sequential( + torch.nn.Linear(X_train.shape[1], 10), + torch.nn.ReLU(), + torch.nn.Linear(10, 3), + torch.nn.Softmax(dim=1) +).to(device) +``` + +We will use the object-oriented API inheriting from `torch.nn.Module` for the remainder of this course, +since it is more flexible and more reusable. + + + +::::::: + +:::::::: +::::::::: ::: callout + ## How to choose an architecture? + Even for this small neural network, we had to make a choice on the number of hidden neurons. Other choices to be made are the number of layers and type of layers (as we will see later). You might wonder how you should make these architectural choices. @@ -547,40 +833,47 @@ Another best practice is to start with a relatively simple architecture. Once ru ::: ### Choose a pretrained model + If your data and problem is very similar to what others have done, you can often use a *pretrained network*. Even if your problem is different, but the data type is common (for example images), you can use a pretrained network and finetune it for your problem. A large number of openly available pretrained networks can be found on [Hugging Face](https://huggingface.co/models) (especially LLMs), [MONAI](https://monai.io/) (medical imaging), the [Model Zoo](https://modelzoo.co/), [pytorch hub](https://pytorch.org/hub/) or [tensorflow hub](https://www.tensorflow.org/hub/). -We will cover the concept of Transfer Learning in [episode 5](./5-transfer-learning.html) +We will cover the concept of Transfer Learning in [episode 5](./5-transfer-learning.md) ## 5. Choose a loss function and optimizer -We have now designed a neural network that in theory we should be able to -train to classify Penguins. -However, we first need to select an appropriate loss -function that we will use during training. -This loss function tells the training algorithm how wrong, or how 'far away' from the true -value the predicted value is. - -For the one-hot encoding that we selected earlier a suitable loss function is the Categorical Crossentropy loss. -In Keras this is implemented in the `keras.losses.CategoricalCrossentropy` class. -This loss function works well in combination with the `softmax` activation function -we chose earlier. -The Categorical Crossentropy works by comparing the probabilities that the -neural network predicts with 'true' probabilities that we generated using the one-hot encoding. -This is a measure for how close the distribution of the three neural network outputs corresponds to the distribution of the three values in the one-hot encoding. -It is lower if the distributions are more similar. - -For more information on the available loss functions in Keras you can check the -[documentation](https://www.tensorflow.org/api_docs/python/tf/keras/losses). + +We have now designed a neural network that in theory we should be able to train +to classify Penguins. However, we first need to select an appropriate loss +function that we will use during training. This loss function tells the +training algorithm how wrong, or how 'far away' from the true value the +predicted value is. + +For the one-hot encoding that we selected earlier a suitable loss function is +the Categorical Crossentropy loss. In Keras this is implemented in the +`keras.losses.CategoricalCrossentropy` class, whereas Pytorch uses +`torch.nn.CrossEntropyLoss`. This loss function works well in combination with +the `softmax` activation function we chose earlier. The Categorical +Crossentropy works by comparing the probabilities that the neural network +predicts with 'true' probabilities that we generated using the one-hot +encoding. This is a measure for how close the distribution of the three neural +network outputs corresponds to the distribution of the three values in the +one-hot encoding. It is lower if the distributions are more similar. + +For more information on the available loss functions in the two frameworks you can check the +documentation for [Keras](https://www.tensorflow.org/api_docs/python/tf/keras/losses) and [Pytorch](https://docs.pytorch.org/docs/stable/nn.html#loss-functions) respectively. Next we need to choose which optimizer to use and, if this optimizer has parameters, what values to use for those. Furthermore, we need to specify how many times to show the training samples to the optimizer. -Once more, Keras gives us plenty of choices all of which have their own pros and cons, +Once more, both frameworks give us plenty of choices all of which have their own pros and cons, but for now let us go with the widely used [Adam optimizer](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam). Adam has a number of parameters, but the default values work well for most problems. So we will use it with its default parameters. +::::::: group-tab + +###### Keras + Combining this with the loss function we decided on earlier we can now compile the model using `model.compile`. Compiling the model prepares it to start the training. @@ -589,10 +882,31 @@ Compiling the model prepares it to start the training. model.compile(optimizer='adam', loss=keras.losses.CategoricalCrossentropy()) ``` + + +###### PyTorch + +In Pytorch, we define the loss function and optimizers. Moreover, we need DataLoaders to feed the train dataset into the neural network. The model is then set in "training mode" using `model.train()`: + +```python + +loss_fn = torch.nn.CrossEntropyLoss() +optimizer = torch.optim.Adam(model.parameters()) + +``` + + +::::::: + ## 6. Train model + We are now ready to train the model. -Training the model is done using the `fit` method, it takes the input data and +::::::: group-tab + +###### Keras + +In Keras, training the model is done using the `fit` method, it takes the input data and target data as inputs and it has several other parameters for certain options of the training. Here we only set a different number of `epochs`. @@ -607,13 +921,64 @@ The fit method returns a history object that has a history attribute with the tr potentially other metrics per training epoch. It can be very insightful to plot the training loss to see how the training progresses. Using seaborn we can do this as follows: + ```python sns.lineplot(x=history.epoch, y=history.history['loss']) ``` + + + +###### PyTorch + +In Pytorch, the training loop has to be defined explicitly: + +```python +train_dataset = torch.utils.data.TensorDataset( + torch.tensor(X_train_scaled, dtype = torch.float), + torch.tensor(y_train.values, dtype = torch.float) +) +train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True) +history = { + 'loss': [] +} +epochs = 100 + +for epoch in range(epochs): + running_loss = 0.0 + + for X_batch, y_batch in train_dataloader: + X_batch, y_batch = X_batch.to(device), y_batch.to(device) + + y_pred = model(X_batch) + loss = loss_fn(y_pred, y_batch) + loss.backward() + optimizer.step() + optimizer.zero_grad() + + running_loss += loss.item() + + train_loss = running_loss / len(train_dataloader) + history['loss'].append(train_loss) + print(f'Epoch {epoch+1:>3d} completed; loss: {train_loss:.4f}') +``` + +The training loss can then be plotted: + +```python +sns.lineplot(x=range(epochs), y=history['loss']) +``` + + + +::::::: + ![][training_curve]{alt="Plot of the Cross Entropy loss, showing a sharp decrease in the first around 10 epochs, and converging at a low value afterwards."} + ::: callout + ## I get a different plot + It could be that you get a different plot than the one shown here. This could be because of a different random initialization of the model or a different split of the data. This difference can be avoided by setting `random_state` and random seed in the same way like we discussed @@ -624,13 +989,15 @@ This plot can be used to identify whether the training is well configured or whe are problems that need to be addressed. :::: challenge + ## The Training Curve + Looking at the training curve we have just made. 1. How does the training progress? - * Does the training loss increase or decrease? - * Does it change quickly or slowly? - * Does the graph look very jittery? + - Does the training loss increase or decrease? + - Does it change quickly or slowly? + - Does the graph look very jittery? 2. Do you think the resulting trained network will work well on the test set? When the training process does not go well: @@ -638,9 +1005,12 @@ When the training process does not go well: 3. (optional) Something went wrong here during training. What could be the problem, and how do you see that in the training curve? Also compare the range on the y-axis with the previous training curve. ![][bad-training-curve] +::: ::: solution + ## Solution + 1. The training loss decreases quickly. It drops in a smooth line with little jitter. This is ideal for a training curve. 2. The results of the training give very little information on its performance on a test set. @@ -659,12 +1029,16 @@ In this case the graph was created by training on nonsense data, so this a train We will take a closer look at training curves in the next episode. Some of the concepts touched upon here will also be further explained there. -::: :::: ## 7. Perform a prediction/classification + Now that we have a trained neural network, we can use it to predict new samples -of penguin using the `predict` function. +of penguins. + +::::::: group-tab + +###### Keras We will use the neural network to predict the species of the test set using the `predict` function. @@ -672,11 +1046,13 @@ We will be using this prediction in the next step to measure the performance of trained network. This will return a `numpy` matrix, which we convert to a pandas dataframe to easily see the labels. + ```python y_pred = model.predict(X_test) prediction = pd.DataFrame(y_pred, columns=target.columns) prediction ``` + | | | | | | --: | -------: | --------: | -------: | | 0 | 0.304484 | 0.192893 | 0.502623 | @@ -691,10 +1067,43 @@ prediction | 67 | 0.393868 | 0.159575 | 0.446557 | | 68 | 0.509837 | 0.144219 | 0.345943 | - Remember that the output of the network uses the `softmax` activation function and has three outputs, one for each species. This dataframe shows this nicely. + + +###### PyTorch + +To run inference (i.e. make predictions) with Pytorch, we need to set the model in "evaluation mode". +Moreover, we wrap the prediction in a `torch.no_grad()` context so that the execution is faster (more on that later). + +```python +model.eval() +with torch.no_grad(): + y_pred = model(torch.tensor(X_test_scaled, dtype=torch.float, device=device)) + +prediction = pd.DataFrame(y_pred.to('cpu'), columns=target.columns) +prediction +``` + +```output + Adelie Chinstrap Gentoo +0 0.786800 0.108887 0.104313 +1 0.874215 0.083492 0.042294 +2 0.903556 0.066898 0.029546 +3 0.126868 0.097586 0.775546 +4 0.628735 0.208464 0.162801 +... ... ... ... +64 0.900963 0.073662 0.025375 +65 0.926656 0.052395 0.020949 +66 0.840132 0.116230 0.043638 +67 0.097195 0.085259 0.817547 +68 0.043282 0.044229 0.912489 +``` + + +::::::: + We now need to transform this output to one penguin species per sample. We can do this by looking for the index of highest valued output and converting that to the corresponding species. @@ -720,13 +1129,15 @@ predicted_species Length: 69, dtype: object ``` - ::: instructor + ## BREAK + This is a good time for switching instructor and/or a break. ::: ## 8. Measuring performance + Now that we have a trained neural network it is important to assess how well it performs. We want to know how well it will perform in a realistic prediction scenario, measuring performance will also come back when refining the model. @@ -735,6 +1146,7 @@ We have created a test set (i.e. y_test) during the data preparation stage which now to create a confusion matrix. ### Confusion matrix + With the predicted species we can now create a confusion matrix and display it using seaborn. A confusion matrix is an `N x N` matrix used for evaluating the performance of a classification model, where `N` is the number of target classes. @@ -753,6 +1165,7 @@ true_species = y_test.idxmax(axis="columns") matrix = confusion_matrix(true_species, predicted_species) print(matrix) ``` + ```output [[22 0 8] [ 5 0 9] @@ -780,6 +1193,7 @@ the heatmap. ```python sns.heatmap(confusion_df, annot=True, cmap='Blues') ``` + ![][confusion_matrix] Here are more explanations of this confusion matrix and the classification model. @@ -789,16 +1203,21 @@ Here are more explanations of this confusion matrix and the classification model - The third row: There are 25 Gentoo penguins in the test data, with 6 identified as Adelie (invalid), none being recognized as Chinstrap (invalid), and 19 Gentoos are identified as Gentoo (valid). :::: challenge + ## Confusion Matrix + Measure the performance of the neural network you trained and visualize a confusion matrix. - Did the neural network perform well on the test set? - Did you expect this from the training loss you saw? - What could we do to improve the performance? +::: ::: solution + ## Solution + The confusion matrix shows that the predictions for Adelie and Gentoo are decent, but could be improved. However, Chinstrap is not predicted ever. If we go back to the [**Pair Plot**](#pair-plot) in the Visualization section above, we can figure out that the biggest challenge is distinguishing the Chinstrap penguins from the marginal distributions of the four features (bill length, bill depth, flipper length, and body mass). That means that there is no single variable that separates Chinstrap penguins from all other species. Only the combination of bill length and bill depth gives a good separation of Chinstrap from Adelie and Gentoo penguins. @@ -810,41 +1229,55 @@ We can try many things to improve the performance from here. One of the first th Furthermore, the constructed neural network has a limited number of parameters. A practical workaround is to increase the number of dense layers and also the number of neurons in each dense layers. -In addition, adjusting the learning rate can also help achieving a high score for the prediction. You will get more info in the [**Advanced layer types**](./4-advanced-layer-types.html) episode. +In addition, adjusting the learning rate can also help achieving a high score for the prediction. You will get more info in the [**Advanced layer types**](./4-advanced-layer-types.md) episode. Note that the outcome you have might be slightly different from what is shown in this tutorial. -::: :::: ## 9. Refine the model + As we discussed before the design and training of a neural network comes with many hyperparameter and model architecture choices. We will go into more depth of these choices in later episodes. For now it is important to realize that the parameters we chose were somewhat arbitrary and more careful consideration needs to be taken to -pick hyperparameter values. - +pick hyperparameter values. ## 10. Share model + It is very useful to be able to use the trained neural network at a later stage without having to retrain it. -This can be done by using the `save` method of the model. + +::::::: group-tab + +###### Keras + +In Keras, this can be done by using the `save` method of the model. It takes a string as a parameter which is the path of a directory where the model is stored. +However, we are also using the `RobustScaler` from `sklearn` to scale the data, thus we also need it in inference. An efficient way to do this is to use the pickler from `joblib`. Thus the whole pipeline can be serialised to disk in the following manner: + +```python +import joblib + +joblib.dump(feature_scaler, 'penguins_scaler.gz') +``` ```python model.save('my_first_model.keras') ``` This saved model can be loaded again by using the `load_model` method as follows: + ```python pretrained_model = keras.models.load_model('my_first_model.keras') +pretrained_scaler = joblib.load('penguins_scaler.gz') ``` This loaded model can be used as before to predict. ```python # use the pretrained model here -y_pretrained_pred = pretrained_model.predict(X_test) +y_pretrained_pred = pretrained_model.predict(pretrained_scaler.transform(X_test)) pretrained_prediction = pd.DataFrame(y_pretrained_pred, columns=target.columns.values) # idxmax will select the column for each row with the highest value @@ -867,7 +1300,46 @@ print(pretrained_predicted_species) Length: 69, dtype: object ``` + + +###### PyTorch +In Pytorch, we can use the `torch.save()` method to save the architecture (and its weights and biases) to disk. +However, we are also using the `RobustScaler` from `sklearn` to scale the data, thus we also need it in inference. An efficient way to do this is to use the pickler from `joblib`. Thus the whole pipeline can be serialised to disk in the following manner: + +```python +import joblib + +joblib.dump(feature_scaler, 'penguins_scaler.gz') +torch.save(model.state_dict(), 'penguins_classification.pt') +``` + +Then we can reload it from disk using the `load_state_dict` method for the network and the `joblib.load()` method for the scaler: + +```python +pretrained_model = PenguinModel(X_train.shape[1]).to(device) +pretrained_model.load_state_dict(torch.load('penguins_classification.pt')) +pretrained_scaler = joblib.load('penguins_scaler.gz') +``` + +The inference is done exactly as before: + +```python +model.eval() +with torch.no_grad(): + X_test_scaled = torch.tensor(pretrained_scaler.transform(X_test), dtype=torch.float, device=device) + y_pretrained_pred = model(X_test_scaled) + +pretrained_prediction = pd.DataFrame(y_pretrained_pred.to('cpu'), columns=target.columns.values) + +# idxmax will select the column for each row with the highest value +pretrained_predicted_species = pretrained_prediction.idxmax(axis="columns") +print(pretrained_predicted_species) +``` + + + +::::::: [palmer-penguins]: fig/palmer_penguins.png "Palmer Penguins" {alt='Illustration of the three species of penguins found in the Palmer Archipelago, Antarctica: Chinstrap, Gentoo and Adele'} @@ -892,14 +1364,15 @@ Length: 69, dtype: object [confusion_matrix]: fig/confusion_matrix.png "Confusion Matrix" {alt='Confusion matrix of the test set with high accuracy for Adelie and Gentoo classification and no correctly predicted Chinstrap'} - :::: keypoints + - The deep learning workflow is a useful tool to structure your approach, it helps to make sure you do not forget any important steps. - Exploring the data is an important step to familiarize yourself with the problem and to help you determine the relavent inputs and outputs. -- One-hot encoding is a preprocessing step to prepare labels for classification in Keras. +- One-hot encoding is a preprocessing step to prepare labels for classification. - A fully connected layer is a layer which has connections to all neurons in the previous and subsequent layers. -- keras.layers.Dense is an implementation of a fully connected layer, you can set the number of neurons in the layer and the activation function used. +- `keras.layers.Dense` and `torch.nn.Linear` are implementations of a fully connected layer, you can set the number of neurons in the layer. In Keras, you can also set the activation function used. - To train a neural network with Keras we need to first define the network using layers and the Model class. Then we can train it using the model.fit function. +- To train a neural network with PyTorch we need to first define a class inheriting from `torch.nn.Module`.Then, we define the layers in the `__init__` method and the forward pass in the `forward` method. Finally, we can train it using a custom training loop. - Plotting the loss curve can be used to identify and troubleshoot the training process. - The loss curve on the training set does not provide any information on how well a network performs in a real setting. - Creating a confusion matrix with results from a test set gives better insight into the network's performance. diff --git a/episodes/3-monitor-the-model.md b/episodes/3-monitor-the-model.md index 7f5511edd..14afe0df4 100644 --- a/episodes/3-monitor-the-model.md +++ b/episodes/3-monitor-the-model.md @@ -63,7 +63,7 @@ import pandas as pd data = pd.read_csv("https://zenodo.org/record/5071376/files/weather_prediction_dataset_light.csv?download=1") ``` -#### SSL certificate error +**SSL certificate error** If you get the following error message: `certificate verify failed: unable to get local issuer certificate`, you can download [the data from here manually](https://zenodo.org/record/5071376/files/weather_prediction_dataset_light.csv?download=1) @@ -196,6 +196,38 @@ In our example we want to predict the sunshine hours in Basel (or any other plac We compose a network of two hidden layers to start off with something. We go by a scheme with 100 neurons in the first hidden layer and 50 neurons in the second layer. As activation function we settle on the `relu` function as a it is very robust and widely used. To make our live easier later, we wrap the definition of the network in a function called `create_nn()`. +::::::: group-tab + +###### PyTorch + +```python +import torch.nn.functional as F +import torch.nn as nn + +class WeatherPredictionModel(nn.Module): + def __init__(self, input_shape, hidden1=100, hidden2=50): + super().__init__() + + self.hidden_layer1 = nn.Linear(input_shape, hidden1) + self.hidden_layer2 = nn.Linear(hidden1, hidden2) + self.output_layer = nn.Linear(hidden2, 1) + + def forward(self, x): + x = self.hidden_layer1(x) + x = F.relu(x) + x = self.hidden_layer2(x) + x = F.relu(x) + x = self.output_layer(x) + return x + +model = WeatherPredictionModel(X_train.shape[1]) + +``` + + + +###### Keras + ```python from tensorflow import keras keras.utils.set_random_seed(2) @@ -216,6 +248,10 @@ def create_nn(input_shape): model = create_nn(input_shape=(X_data.shape[1],)) ``` + + +::::::: + The shape of the input layer has to correspond to the number of features in our data: `89`. We use `X_data.shape[1]` to obtain this value dynamically The output layer here is a dense layer with only 1 node. And we here have chosen to use *no activation function*. @@ -223,7 +259,56 @@ While we might use *softmax* for a classification task, here we do not want to r In addition, we have here chosen to write the network creation as a function so that we can use it later again to initiate new models. -Let us check how our model looks like by calling the `summary` method. +Let us check how our model looks like by using a `summary` method. + +::::::: group-tab + +###### PyTorch + +```python +print(model) +``` +```output + + +WeatherPredictionModel( + (hidden_layer1): Linear(in_features=89, out_features=100, bias=True) + (hidden_layer2): Linear(in_features=100, out_features=50, bias=True) + (output_layer): Linear(in_features=50, out_features=1, bias=True) +) + +``` + +```python +from torchinfo import summary + +example_batch_size = 32 +summary(model, input_size=(example_batch_size, X_train.shape[1])) +``` +```output +========================================================================================== +Layer (type:depth-idx) Output Shape Param # +========================================================================================== +WeatherPredictionModel [32, 1] -- +├─Linear: 1-1 [32, 100] 9,000 +├─Linear: 1-2 [32, 50] 5,050 +├─Linear: 1-3 [32, 1] 51 +========================================================================================== +Total params: 14,101 +Trainable params: 14,101 +Non-trainable params: 0 +Total mult-adds (Units.MEGABYTES): 0.45 +========================================================================================== +Input size (MB): 0.01 +Forward/backward pass size (MB): 0.04 +Params size (MB): 0.06 +Estimated Total Size (MB): 0.11 +========================================================================================== +``` + + + +###### Keras ```python model.summary() @@ -250,9 +335,13 @@ Model: "weather_prediction_model" Non-trainable params: 0 (0.00 B) ``` - When compiling the model we can define a few very important aspects. We will discuss them now in more detail. + + +::::::: + + ## Intermezzo: How do neural networks learn? In the introduction we learned about the loss function: it quantifies the total error of the predictions made by the model. During model training we aim to find the model parameters that minimize the loss. @@ -284,25 +373,26 @@ This subset is called a 'batch', the number of samples in one batch is called th Answer the following questions: -### 1. What is the goal of optimization? +**1. What is the goal of optimization?** - A. To find the weights that maximize the loss function - B. To find the weights that minimize the loss function -### 2. What happens in one gradient descent step? +**2. What happens in one gradient descent step?** - A. The weights are adjusted so that we move in the direction of the gradient, so up the slope of the loss function - B. The weights are adjusted so that we move in the direction of the gradient, so down the slope of the loss function - C. The weights are adjusted so that we move in the direction of the negative gradient, so up the slope of the loss function - D. The weights are adjusted so that we move in the direction of the negative gradient, so down the slope of the loss function -### 3. When the batch size is increased: +**3. When the batch size is increased:** (multiple answers might apply) - A. The number of samples in an epoch also increases - B. The number of batches in an epoch goes down - C. The training progress is more jumpy, because more samples are consulted in each update step (one batch). - D. The memory load (memory as in computer hardware) of the training process is increased +::: ::: solution @@ -320,13 +410,24 @@ Answer the following questions: - C. The training progress is more jumpy, because more samples are consulted in each update step (one batch). (**incorrect**, more samples are consulted in each update step, but this makes the progress less jumpy since you get a more accurate estimate of the loss in the entire dataset) - D. The memory load (memory as in computer hardware) of the training process is increased (**correct**, the data is begin loaded one batch at a time, so more samples means more memory usage) -::: :::: ## 5. Choose a loss function and optimizer ### Loss function The loss is what the neural network will be optimized on during training, so choosing a suitable loss function is crucial for training neural networks. In the given case we want to stimulate that the predicted values are as close as possible to the true values. This is commonly done by using the *mean squared error* (mse) or the *mean absolute error* (mae), both of which should work OK in this case. Often, mse is preferred over mae because it "punishes" large prediction errors more severely. + +::::::: group-tab + +###### PyTorch + +In PyTorch the is implemented in the `torch.nn.MSELoss` class (see PyTorch documentation: https://docs.pytorch.org/docs/stable/nn.html#loss-functions). + + + + +###### Keras + In Keras this is implemented in the `keras.losses.MeanSquaredError` class (see Keras documentation: https://keras.io/api/losses/). This can be provided into the `model.compile` method with the `loss` parameter and setting it to `mse`, e.g. @@ -334,18 +435,38 @@ In Keras this is implemented in the `keras.losses.MeanSquaredError` class (see K model.compile(loss='mse') ``` + + +::::::: ### Optimizer Somewhat coupled to the loss function is the *optimizer* that we want to use. The *optimizer* here refers to the algorithm with which the model learns to optimize on the provided loss function. A basic example for such an optimizer would be *stochastic gradient descent*. For now, we can largely skip this step and pick one of the most common optimizers that works well for most tasks: the *Adam optimizer*. Similar to activation functions, the choice of optimizer depends on the problem you are trying to solve, your model architecture and your data. *Adam* is a good starting point though, which is why we chose it. + +::::::: group-tab + +###### PyTorch + +```python +optimizer = optim.Adam(model.parameters()) +``` + + + +###### Keras + ```python model.compile(optimizer='adam', loss='mse') ``` + + +::::::: + ### Metrics In our first example (episode 2) we plotted the progression of the loss during training. @@ -354,8 +475,27 @@ However, when models become more complicated then also the loss functions often That is why it is good practice to monitor the training process with additional, more intuitive metrics. They are not used to optimize the model, but are simply recorded during training. +::::::: group-tab + +###### PyTorch + +In pure PyTorch any additional metrics can be computed from the predictions during the training process. Here we import [`metrics`](https://scikit-learn.org/stable/api/sklearn.metrics.html#module-sklearn.metrics) from `sklearn` to avoid implementing and testing them ourselves. Another popular option is [`torchmetrics`](https://lightning.ai/docs/torchmetrics/stable/). +Here we could for instance choose [mean absolute error](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error) or [*root mean squared error*](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.root_mean_squared_error.html#sklearn.metrics.root_mean_squared_error) which unlike the *MSE* have the same units as the predicted values. We choose the latter. + +```python +from sklearn import metrics +for i, batch in enumerate(data_loader): + # train + running_rmse += metrics.root_mean_squared_error(y_batch, y_pred) +train_rmse = running_rmse / len(data_loader) +``` + + + +###### Keras + With Keras, such additional metrics can be added via `metrics=[...]` parameter and can contain one or multiple metrics of interest. -Here we could for instance chose `mae` ([mean absolute error](https://glosario.carpentries.org/en/#mean_absolute_error)), or the the [*root mean squared error* (RMSE)](https://glosario.carpentries.org/en/#root_mean_squared_error) which unlike the *mse* has the same units as the predicted values. For the sake of units, we choose the latter. +Here we could for instance choose `mae` ([mean absolute error](https://glosario.carpentries.org/en/#mean_absolute_error)), or the [*root mean squared error* (RMSE)](https://glosario.carpentries.org/en/#root_mean_squared_error) which unlike the *mse* have the same units as the predicted values. We choose the latter. ```python model.compile(optimizer='adam', @@ -374,12 +514,119 @@ compile_model(model) With this, we complete the compilation of our network and are ready to start training. + + +::::::: + ## 6. Train the model Now that we created and compiled our dense neural network, we can start training it. We add the `batch_size` parameter that defines -- as discussed above -- how many samples from the training data will be used to estimate the error gradient before the model weights are updated. Larger batches will produce better, more accurate gradient estimates but also less frequent updates of the weights. Here we are going to use a batch size of 32 which is a common starting point. + +::::::: group-tab + +###### PyTorch + +In pure PyTorch, we can write a short training loop to compute the prediction of our model for each batch in the `data_loader`. With the `prediction`, the true `label`, and the `loss function` we can compute the `loss value`. With the `loss value`, the `optimizer` can then perform backpropagation. + +```python +def train_epoch(model, data_loader, loss_fn, optimizer, progress_desc): + """Training the model for one epoch.""" + model.train() + running_loss = 0.0 + running_rmse = 0.0 + + if progress_desc: + pbar = tqdm(data_loader, desc=progress_desc) + else: + pbar = data_loader + + for i, (X_batch, y_batch) in enumerate(pbar): + X_batch = X_batch.to(device) + y_batch = y_batch.to(device) + + # Zero the gradients + optimizer.zero_grad() + + # Forward pass and make predictions + y_pred = model(X_batch) + + # Compute loss + loss = loss_fn(y_pred.squeeze(), y_batch) + + # Backward pass and update weights using the optimizer + loss.backward() + optimizer.step() + + running_loss += loss.item() + running_rmse += metrics.root_mean_squared_error(y_batch, y_pred) + + if progress_desc: + pbar.set_postfix({"loss": running_loss / (i + 1), "rmse": running_rmse / (i + 1)}) + + train_loss = running_loss / len(data_loader) + train_rmse = running_rmse / len(data_loader) + return train_loss, train_rmse +``` + +We should also regularly evaluate our model on the validation dataset. +We could e.g. do this after a certain number of training iterations. +Here we choose to do it after every completed epoch. + +```python +def eval_epoch(model, data_loader, loss_fn, accumulate=False): + """Evaluate the model for one epoch for testing / validation data. + No gradients are computed, no backpropagation. + + """ + model.eval() + running_loss = 0.0 + running_rmse = 0.0 + y_true_tensor = torch.tensor([]).to(device) + y_pred_tensor = torch.tensor([]).to(device) + + with torch.no_grad(): + for X_batch, y_batch in data_loader: + X_batch = X_batch.to(device) + y_batch = y_batch.to(device) + + y_pred = model(X_batch) + loss = loss_fn(y_pred.squeeze(), y_batch) + + running_loss += loss.item() + running_rmse += metrics.root_mean_squared_error(y_batch, y_pred) + + if accumulate: + y_true_tensor = torch.cat((y_true_tensor, y_batch)) + y_pred_tensor = torch.cat((y_pred_tensor, y_pred)) + + eval_loss = running_loss / len(data_loader) + eval_rmse = running_rmse / len(data_loader) + return eval_loss, eval_rmse, y_true_tensor.cpu(), y_pred_tensor.cpu() +``` + +To visualize the training process, we keep track of these metrics in a `history` `dict`: +```python + + +model = model.to(device) + +history = {'loss': [], 'root_mean_squared_error': []} +epochs = 200 + +for epoch in range(epochs): + loss, rmse = train_epoch(model, train_dl, loss_fn, optimizer, f"Epoch {epoch+1}/{epochs}") + history['loss'].append(loss) + history['root_mean_squared_error'].append(rmse) + +``` + + + +###### Keras + ```python history = model.fit(X_train, y_train, batch_size=32, @@ -387,8 +634,37 @@ history = model.fit(X_train, y_train, verbose=2) ``` + + +::::::: + We can plot the training process using the `history` object returned from the model training. We will create a function for it, because we will make use of this more often in this lesson! + +::::::: group-tab + +###### PyTorch + +```python +def plot_history(history, metrics): + """ + Plot the training history + + Args: + history (dict): Dictionary containing training history + metrics (str, list): Metric or a list of metrics to plot + """ + history_df = pd.DataFrame(history) + sns.lineplot(data=history_df[metrics]) + plt.xlabel("epochs") + plt.ylabel("metric") + +plot_history(history, 'root_mean_squared_error') +``` + + + +###### Keras ```python import seaborn as sns import matplotlib.pyplot as plt @@ -409,6 +685,10 @@ def plot_history(history, metrics): plot_history(history, 'root_mean_squared_error') ``` + + +::::::: + ![](fig/03_training_history_1_rmse.png){alt='Plot of the RMSE over epochs for the trained model that shows a decreasing error metric.'} This looks very promising! Our metric "root_mean_squared_error" (RMSE) is dropping nicely and while it maybe keeps fluctuating a bit it does end up at fairly low values. @@ -417,11 +697,28 @@ But this metric is just the root *mean* squared error, so we might want to look ## 7. Perform a Prediction/Classification Now that we have our model trained, we can make a prediction with the model before measuring the performance of our neural network. +::::::: group-tab + +###### PyTorch + +```python +_, train_rmse, y_train_true, y_train_predicted = eval_epoch(model, train_dl, loss_fn, accumulate=True) +_, test_rmse, y_test_true, y_test_predicted = eval_epoch(model, test_dl, loss_fn, accumulate=True) +``` + + + +###### Keras + ```python y_train_predicted = model.predict(X_train) y_test_predicted = model.predict(X_test) ``` + + +::::::: + ::: instructor ## BREAK This is a good time for switching instructor and/or a break. @@ -444,15 +741,49 @@ def plot_predictions(y_pred, y_true, title): plt.xlabel("predicted sunshine hours") plt.ylabel("true sunshine hours") plt.title(title) +``` + +::::::: group-tab + +###### PyTorch + +```python +plot_predictions(y_train_predicted, y_train_true, title='Predictions on the training set') +``` + + + + +###### Keras +```python plot_predictions(y_train_predicted, y_train, title='Predictions on the training set') ``` + + +::::::: + ![](fig/03_regression_predictions_trainset.png){alt='Scatter plot between predictions and true sunshine hours in Basel on the training set showing a concise spread'} +::::::: group-tab + +###### PyTorch +```python +plot_predictions(y_test_predicted, y_test_true, title='Predictions on the test set') +``` + + + +###### Keras + ```python plot_predictions(y_test_predicted, y_test, title='Predictions on the test set') ``` + + +::::::: + ![](fig/03_regression_predictions_testset.png){alt='Scatter plot between predictions and true sunshine hours in Basel on the test set showing a wide spread'} :::: challenge @@ -469,7 +800,8 @@ plot_predictions(y_test_predicted, y_test, title='Predictions on the test set') While the performance on the train set seems reasonable, the performance on the test set is much worse. This is a common problem called **overfitting**, which we will discuss in more detail later. -#### Optional exercise: +**Optional exercise:** + The metric that we are using: RMSE would be a good one. You could also consider Mean Squared Error, that punishes large errors more (because large errors create even larger squared errors). It is important that if the model improves in performance on the basis of this metric then that should also lead you a step closer to reaching your goal: to predict tomorrow's sunshine hours. If you feel that improving the metric does not lead you closer to your goal, then it would be better to choose a different metric @@ -481,6 +813,21 @@ In fact, considering that the task of predicting the daily sunshine hours is rea (at least on the training set). Maybe a little too good? We also see the noticeable difference between train and test set when calculating the exact value of the RMSE: +::::::: group-tab + +###### PyTorch + +```python +print(f'Train RMSE: {train_rmse:.2f}, Test RMSE: {test_rmse:.2f}') +``` +```output +Train RMSE: 0.91, Test RMSE: 4.19 +``` + + + +###### Keras + ```python train_metrics = model.evaluate(X_train, y_train, return_dict=True) test_metrics = model.evaluate(X_test, y_test, return_dict=True) @@ -492,6 +839,10 @@ print('Train RMSE: {:.2f}, Test RMSE: {:.2f}'.format(train_metrics['root_mean_sq Train RMSE: 0.84, Test RMSE: 4.05 ``` + + +::::::: + For those experienced with (classical) machine learning this might look familiar. The plots above expose the signs of **overfitting** which means that the model has to some extent memorized aspects of the training data. As a result, it makes much more accurate predictions on the training data than on unseen test data. @@ -565,7 +916,49 @@ set can be used during training, and the test set is reserved for afterwards. Let's give this a try! -We need to initiate a new model -- otherwise Keras will simply assume that we want to continue training the model we already trained above. +::::::: group-tab + +###### PyTorch + +We need to initialize a new model -- otherwise we would continue training the parameters we already trained above. +We also need to create a validation `dataset` and `dataloader`. + +```python +val_dataset = TensorDataset( + torch.tensor(X_val.values, dtype=torch.float), + torch.tensor(y_val.values, dtype=torch.float) +) +val_dl = DataLoader(val_dataset, batch_size=32, shuffle=False) + +model = WeatherPredictionModel(input_shape=X_data.shape[1]) +optimizer = optim.Adam(model.parameters()) +loss_fn = nn.MSELoss() + +model = model.to(device) + +history = { + 'loss': [], + 'root_mean_squared_error': [], + 'val_loss': [], + 'val_root_mean_squared_error': [] +} +epochs = 200 + +for epoch in range(epochs): + loss, rmse = train_epoch(model, train_dl, loss_fn, optimizer, False) + history['loss'].append(loss) + history['root_mean_squared_error'].append(rmse) + + val_loss, val_rmse, *_ = eval_epoch(model, val_dl, loss_fn) + history['val_loss'].append(val_loss) + history['val_root_mean_squared_error'].append(val_rmse) +``` + + + +###### Keras + +We need to initialize a new model -- otherwise Keras will simply assume that we want to continue training the model we already trained above. ```python model = create_nn(input_shape=(X_data.shape[1],)) compile_model(model) @@ -578,6 +971,9 @@ history = model.fit(X_train, y_train, epochs=200, validation_data=(X_val, y_val)) ``` + + +::::::: With this we can plot both the performance on the training data and on the validation data! @@ -611,7 +1007,7 @@ to 0. After that the curves will just consistently stay at 0. Overfitting is a very common issue and there are many strategies to handle it. Most similar to classical machine learning might to **reduce the number of parameters**. -:::: challenge +:::::::::::::::::::: challenge ## Exercise: Try to reduce the degree of overfitting by lowering the number of parameters We can keep the network architecture unchanged (2 dense layers + a one-node output layer) and only play with the number of nodes per layer. Try to lower the number of nodes in one or both of the two dense layers and observe the changes to the training and validation losses. @@ -621,9 +1017,73 @@ If time is short: Suggestion is to run one network with only 10 and 5 nodes in t 2. Does the overall performance suffer or does it mostly stay the same? 3. (optional) How low can you go with the number of parameters without notable effect on the performance on the validation set? -::: solution +:::::::::::::::::: solution ## Solution +::::::: group-tab + +###### PyTorch + +```python +model = WeatherPredictionModel(input_shape=X_data.shape[1], hidden1=10, hidden2=5) +summary(model, input_size=(32, X_data.shape[1])) +``` +```output +========================================================================================== +Layer (type:depth-idx) Output Shape Param # +========================================================================================== +WeatherPredictionModel [32, 1] -- +├─Linear: 1-1 [32, 10] 900 +├─Linear: 1-2 [32, 5] 55 +├─Linear: 1-3 [32, 1] 6 +========================================================================================== +Total params: 961 +Trainable params: 961 +Non-trainable params: 0 +Total mult-adds (Units.MEGABYTES): 0.03 +========================================================================================== +Input size (MB): 0.01 +Forward/backward pass size (MB): 0.00 +Params size (MB): 0.00 +Estimated Total Size (MB): 0.02 +========================================================================================== +``` + +Let's train this network: + +```python + + +optimizer = optim.Adam(model.parameters()) +loss_fn = nn.MSELoss() + + +model = model.to(device) + +history = { + 'loss': [], + 'root_mean_squared_error': [], + 'val_loss': [], + 'val_root_mean_squared_error': [] +} +epochs = 200 + +for epoch in range(epochs): + loss, rmse = train_epoch(model, train_dl, loss_fn, optimizer, False) + history['loss'].append(loss) + history['root_mean_squared_error'].append(rmse) + + val_loss, val_rmse, *_ = eval_epoch(model, val_dl, loss_fn) + history['val_loss'].append(val_loss) + history['val_root_mean_squared_error'].append(val_rmse) + +plot_history(history, ['root_mean_squared_error', 'val_root_mean_squared_error']) +``` + + + +###### Keras + Let's first adapt our `create_nn()` function so that we can tweak the number of nodes in the 2 layers by passing arguments to the function: @@ -680,6 +1140,10 @@ history = model.fit(X_train, y_train, plot_history(history, ['root_mean_squared_error', 'val_root_mean_squared_error']) ``` + + +::::::: + ![](fig/03_training_history_3_rmse_smaller_model.png){alt='Plot of RMSE vs epochs for the training set and the validation set with similar performance across the two sets. RMSE for the validation set diverges from RMSE for the training set after around 10 epochs but the difference in RMSE values for the two sets is much smaller than in the previous example.'} 1. With this smaller model we have reduced overfitting a bit, since the training and validation loss are now closer to each other, and the validation loss does now reach a plateau and does not further increase. @@ -687,8 +1151,8 @@ We have not completely avoided overfitting though. 2. In the case of this small example model, the validation RMSE seems to end up around 3.2, which is much better than the 4.08 we had before. Note that you can double check the actual score by calling `model.evaluate()` on the test set. 3. In general, it quickly becomes a complicated search for the right "sweet spot", i.e. the settings for which overfitting will be (nearly) avoided but the model still performs equally well. A model with 3 neurons in both layers seems to be around this spot, reaching an RMSE of 3.1 on the validation set. Reducing the number of nodes further increases the validation RMSE again. -::: -:::: +:::::::::::::::::: +:::::::::::::::::::: We saw that reducing the number of parameters can be a strategy to avoid overfitting. In practice, however, this is usually not the (main) way to go when it comes to deep learning. @@ -701,12 +1165,74 @@ More specifically, this usually means that the training is stopped if the valida Early stopping is both intuitive and effective to use, so it has become a standard addition for model training. To better study the effect, we can now safely go back to models with many (too many?) parameters: + +::::::: group-tab + +###### PyTorch + +```python +model = WeatherPredictionModel(input_shape=X_data.shape[1]) +optimizer = optim.Adam(model.parameters()) +loss_fn = nn.MSELoss() + +def fit(model, train_dl, loss_fn, optimizer, val_dl): + model = model.to(device) + + history = { + 'loss': [], + 'root_mean_squared_error': [], + 'val_loss': [], + 'val_root_mean_squared_error': [] + } + epochs = 200 + early_stopping_patience = 10 + + best_val_loss = float('inf') + patience_counter = 0 + + for epoch in range(epochs): + loss, rmse = train_epoch( + model, + train_dl, + loss_fn, + optimizer, + f"Epoch {epoch+1}/{epochs} {best_val_loss=:.2f}" + ) + history['loss'].append(loss) + history['root_mean_squared_error'].append(rmse) + + val_loss, val_rmse, *_ = eval_epoch(model, val_dl, loss_fn) + history['val_loss'].append(val_loss) + history['val_root_mean_squared_error'].append(val_rmse) + + # Early stopping + if val_loss < best_val_loss: + best_val_loss = val_loss + patience_counter = 0 + # Save best model + best_model_state = model.state_dict() + else: + patience_counter += 1 + if patience_counter >= early_stopping_patience: + print(f'Early stopping at epoch {epoch+1}') + # Restore best model + model.load_state_dict(best_model_state) + break + + return model, history + +model, history = fit(model, train_dl, loss_fn, optimizer, val_dl) +``` + + + +###### Keras ```python model = create_nn(input_shape=(X_data.shape[1],)) compile_model(model) ``` -To apply early stopping during training it is easiest to use Keras `EarlyStopping` class. +To apply early stopping during training it is easiest to use Keras' `EarlyStopping` class. This allows to define the condition of when to stop training. In our case we will say when the validation loss is lowest. However, since we have seen some fluctuation of the losses during training above we will also set `patience=10` which means that the model will stop training if the validation loss has not gone down for 10 epochs. ```python @@ -724,6 +1250,10 @@ history = model.fit(X_train, y_train, callbacks=[earlystopper]) ``` + + +::::::: + As before, we can plot the losses during training: ```python plot_history(history, ['root_mean_squared_error', 'val_root_mean_squared_error']) @@ -745,12 +1275,86 @@ Techniques to avoid overfitting, or to improve model generalization, are termed A very common step in classical machine learning pipelines is to scale the features, for instance by using sckit-learn's `StandardScaler`. This can in principle also be done for deep learning. + +::::::: group-tab + +###### PyTorch +An alternative, more common approach, is to add **BatchNormalization** layers ([documentation of the batch normalization layers](https://keras.io/api/layers/normalization_layers/batch_normalization/)) which will learn how to scale the input values. + + + +###### Keras An alternative, more common approach, is to add **BatchNormalization** layers ([documentation of the batch normalization layer](https://keras.io/api/layers/normalization_layers/batch_normalization/)) which will learn how to scale the input values. + + +::::::: + Similar to dropout, batch normalization is available as a network layer in Keras and can be added to the network in a similar way. It does not require any additional parameter setting. -The `BatchNormalization` can be inserted as yet another layer into the architecture. +Batch normalization can be inserted as yet another layer into the architecture. + +::::::: group-tab + +###### PyTorch +```python +class WeatherPredictionModelBatchNorm(nn.Module): + def __init__(self, input_shape, hidden1=100, hidden2=50): + super().__init__() + self.batch_norm = nn.BatchNorm1d(input_shape) + self.hidden_layer1 = nn.Linear(input_shape, hidden1) + self.hidden_layer2 = nn.Linear(hidden1, hidden2) + self.output_layer = nn.Linear(hidden2, 1) + + def forward(self, x): + x = self.batch_norm(x) + x = self.hidden_layer1(x) + x = F.relu(x) + x = self.hidden_layer2(x) + x = F.relu(x) + x = self.output_layer(x) + return x + +model = WeatherPredictionModel(X_train.shape[1]) +summary(model, input_size=(32, X_train.shape[1])) +``` + +This new layer appears in the model summary as well. + +```output +========================================================================================== +Layer (type:depth-idx) Output Shape Param # +========================================================================================== +WeatherPredictionModel [32, 1] -- +├─Linear: 1-1 [32, 100] 9,000 +├─Linear: 1-2 [32, 50] 5,050 +├─Linear: 1-3 [32, 1] 51 +========================================================================================== +Total params: 14,101 +Trainable params: 14,101 +Non-trainable params: 0 +Total mult-adds (Units.MEGABYTES): 0.45 +========================================================================================== +Input size (MB): 0.01 +Forward/backward pass size (MB): 0.04 +Params size (MB): 0.06 +Estimated Total Size (MB): 0.11 +========================================================================================== +``` + +We can train the model again as follows: + +```python +optimizer = optim.Adam(model.parameters()) +loss_fn = nn.MSELoss() + +model, history = fit(model, train_dl, loss_fn, optimizer, val_dl) +``` + + + +###### Keras ```python def create_nn(input_shape): # Input layer @@ -809,6 +1413,9 @@ history = model.fit(X_train, y_train, plot_history(history, ['root_mean_squared_error', 'val_root_mean_squared_error']) ``` + + +::::::: ![](fig/03_training_history_5_rmse_batchnorm.png){alt='Plot of error vs epochs for the training set and the validation set displaying similar performance across the two sets. RMSE for the validation set drops more than for the training set at first, tracks the training error until approximately 50 epochs, then begins to gradually increase while error for the training set continues to gradually decrease.'} @@ -827,17 +1434,35 @@ The additional parameters gamma and beta are introduced to allow for more flexib It seems that no matter what we add, the overall loss does not decrease much further (we at least avoided overfitting though!). Let us again plot the results on the test set: + + +::::::: group-tab + +###### PyTorch + +```python +_, test_rmse, y_test_true, y_test_predicted = eval_epoch(model, test_dl, loss_fn, accumulate=True) +plot_predictions(y_test_predicted, y_test_true, title='Predictions on the test set') +``` + + + +###### Keras + ```python y_test_predicted = model.predict(X_test) plot_predictions(y_test_predicted, y_test, title='Predictions on the test set') ``` + + +::::::: ![](fig/03_regression_test_5_dropout_batchnorm.png){alt='Scatter plot between predictions and true sunshine hours for Basel on the test set, showing a loose positive correlation.'} Well, the above is certainly not perfect. But how good or bad is this? Maybe not good enough to plan your picnic for tomorrow. But let's better compare it to the naive baseline we created in the beginning. What would you say, did we improve on that? -:::: challenge +:::::::::::::::::::: challenge ## Exercise: Simplify the model and add data You may have been wondering why we are including weather observations from multiple cities to predict sunshine hours only in Basel. The weather is @@ -864,7 +1489,7 @@ but what happens if we limit ourselves to only one city? and all features from all cities. How does it perform? -::: solution +:::::::::::::::::: solution ## Solution ### 1. Use 9 years out of the dataset ```python @@ -889,6 +1514,50 @@ X_train, X_holdout, y_train, y_holdout = train_test_split(X_data, y_data, test_s X_val, X_test, y_val, y_test = train_test_split(X_holdout, y_holdout, test_size=0.5, random_state=0) ``` + +::::::: group-tab + +###### PyTorch + +```python +train_dataset = TensorDataset( + torch.tensor(X_train.values, dtype=torch.float), + torch.tensor(y_train.values, dtype=torch.float) +) +test_dataset = TensorDataset( + torch.tensor(X_test.values, dtype=torch.float), + torch.tensor(y_test.values, dtype=torch.float) +) +val_dataset = TensorDataset( + torch.tensor(X_val.values, dtype=torch.float), + torch.tensor(y_val.values, dtype=torch.float) +) + +train_dl = DataLoader(train_dataset, batch_size=32, shuffle=True) +test_dl = DataLoader(test_dataset, batch_size=32, shuffle=False) +val_dl = DataLoader(val_dataset, batch_size=32, shuffle=False) + +model = WeatherPredictionModelBatchNorm(input_shape=X_data.shape[1]) +summary(model, (32, X_train.shape[1])) +``` + +Fit with early stopping: +```python +optimizer = optim.Adam(model.parameters()) +loss_fn = nn.MSELoss() + +model, history = fit(model, train_dl, loss_fn, optimizer, val_dl) +``` + +Perform predictions: +```python +_, test_rmse, y_test_true, y_test_predicted = eval_epoch(model, test_dl, loss_fn, accumulate=True) +``` + + + +###### Keras + Create the network. We can re-use the `create_nn()` function that we already have. Because we have reduced the number of input features the number of parameters in the network goes down from 14457 to 6137. ```python @@ -909,14 +1578,40 @@ history = model.fit(X_train, y_train, plot_history(history, ['root_mean_squared_error', 'val_root_mean_squared_error']) ``` -Create a scatter plot to compare with true observations: +Perform predictions: ```python y_test_predicted = model.predict(X_test) +``` + + + +::::::: + +Create a scatter plot to compare with true observations: + +```python plot_predictions(y_test_predicted, y_test, title='Predictions on the test set') ``` ![](fig/03_scatter_plot_basel_model.png){alt='Scatterplot of predictions and true number of sunshine hours for all cities, showing many data points distributed in a very loose positive correlation.'} +::::::: group-tab + +###### PyTorch + +```python +print('Baseline:', rmse_baseline) +print('Test RMSE:', test_rmse) +``` +```output +Baseline: 3.877323350410224 +Test RMSE: 3.3969762325286865 +``` + + + +###### Keras + Compute the RMSE on the test set: ```python test_metrics = model.evaluate(X_test, y_test, return_dict=True) @@ -926,6 +1621,10 @@ print(f'Test RMSE: {test_metrics["root_mean_squared_error"]}') Test RMSE: 3.3761725425720215 ``` + + +::::::: + This RMSE is already a lot better compared to what we had before and certainly better than the baseline. Additionally, it could be further improved with hyperparameter tuning. @@ -948,14 +1647,61 @@ For the rest you can use the same code as above to train and evaluate the model This results in an RMSE on the test set of 3.23 (your result can be different, but should be in the same range). From this we can conclude that adding more training data results in even better performance! -::: -:::: +:::::::::::::::::: +:::::::::::::::::::: -::: callout +:::::::::::::::::: callout ## Tensorboard If we run many different experiments with different architectures, it can be difficult to keep track of these different models or compare the achieved performance. We can use *tensorboard*, a framework that keeps track of our experiments and shows graphs like we plotted above. + + +::::::: group-tab + +###### PyTorch + +Tensorboard is included in recent PyTorch versions by default. +To use it, we can create a `SummaryWriter` we can then log metrics to. + +```python +# TensorBoard logging +from datetime import datetime +from torch.utils.tensorboard import SummaryWriter + + +log_dir = "logs/fit/" + datetime.now().strftime("%Y%m%d-%H%M%S") +writer = SummaryWriter(log_dir=log_dir) + +def fit_with_tensorboard(model, train_loader, val_loader, loss_fn, optimizer): + model = model.to(device) + val_loss = float("inf") + epochs = 50 + + for epoch in range(epochs): + # Training phase + loss, rmse = train_epoch( + model, + train_dl, + loss_fn, + optimizer, + f"Epoch {epoch+1}/{epochs} {val_loss=:.2f}" + ) + writer.add_scalar('training loss', loss, epoch) + + # Validation phase + val_loss, val_rmse, *_ = eval_epoch(model, val_loader, loss_fn) + + # Log to TensorBoard + writer.add_scalar('validation loss', val_loss, epoch) + + writer.close() + return model +``` + + + +###### Keras Tensorboard is included in our tensorflow installation by default. To use it, we first need to add a *callback* to our (compiled) model that saves the progress of training performance in a logs rectory: ```python @@ -970,6 +1716,10 @@ history = model.fit(X_train, y_train, callbacks=[tensorboard_callback], verbose = 2) ``` + + +::::::: + You can launch the tensorboard interface from a Jupyter notebook, showing all trained models: ``` @@ -978,15 +1728,32 @@ You can launch the tensorboard interface from a Jupyter notebook, showing all tr ``` Which will show an interface that looks something like this: ![](fig/03_tensorboard.png){alt='Tensorboard graphical user interface.'} -::: +:::::::::::::::::: ## 10. Save model Now that we have a somewhat acceptable model, let us not forget to save it for future users to benefit from our explorative efforts! + +::::::: group-tab + +###### PyTorch + +```python +trained_model_path = "weather_prediction_model_pytorch.pt" +torch.save(model_final.state_dict(), trained_model_path) +``` + + + +###### Keras + ```python model.save('my_tuned_weather_model.keras') ``` + + +::::::: ## Outlook Correctly predicting tomorrow's sunshine hours is apparently not that simple. diff --git a/episodes/4-advanced-layer-types.md b/episodes/4-advanced-layer-types.md index 5b4946a9f..8776639e3 100644 --- a/episodes/4-advanced-layer-types.md +++ b/episodes/4-advanced-layer-types.md @@ -111,12 +111,13 @@ How many features does one image in the Dollar Street 10 dataset have? - C. 12288 - D. 878 +::: ::: solution The correct solution is C: 12288 There are 4096 pixels in one image (64 * 64), each pixel has 3 channels (RGB). So 4096 * 3 = 12288. -::: + :::: @@ -150,14 +151,49 @@ The values of the labels range between `0` and `9`, denoting 10 different classe ## 3. Prepare data -The training set consists of 878 images of `64x64` pixels and 3 channels (RGB values). The RGB values are between `0` and `255`. For input of neural networks, it is better to have small input values. So we normalize our data between `0` and `1`: +The training set consists of 878 images of `64x64` pixels and 3 channels (RGB values). The RGB values are between `0` and `255`. For input of neural networks, it is better to have small input values. So we normalize our data between `0` and `1`. +::::::: group-tab + +###### Keras ```python train_images = train_images / 255.0 val_images = val_images / 255.0 ``` + + +###### PyTorch + +In PyTorch, one common pattern to load and transform complex data is to define a custom `Dataset` class. The class must define two methods: the `__len__` method must return how many samples are in the dataset, while the `__getitem__` method must return the sample corresponding to a specific id. These methods will be automatically called by the dataloader during training or evaluation to build each individual batch. + +In previous lessons, we have used the `TensorDataset` class, which is a built-in type of `Dataset` designed for tabular data. + +In this case, we utilize the constructor of the dataset to load images and labels for the correct data split (train or test). In `__len__`, we simply return how many images are in the split. In `__getitem__`, we return the correct image and label. We convert both the image and label to PyTorch tensors (as we did in the input to the `TensorDataset`s in previous lessons) and we permute the dimensions of the image from `64x64x3` to `3x64x64`, as this is the expected convention in PyTorch. + +```python +class DollarStreetDataset(torch.utils.data.Dataset): + def __init__(self, root, train=True): + prefix = "test" + if train == True: + prefix = "train" + self.images = np.load(root / f'{prefix}_images.npy') / 255. + self.labels = np.load(root / f'{prefix}_labels.npy') + + def __getitem__(self, idx): + x = torch.permute(torch.tensor(self.images[idx], dtype=torch.float), (2, 0, 1)) + y = torch.tensor(self.labels[idx], dtype=torch.long) + return x, y + + def __len__(self): + return self.images.shape[0] +``` + + +::::::: + + ## 4. Choose a pretrained model or start building architecture from scratch ### Convolutional layers @@ -165,8 +201,11 @@ In the previous episodes, we used 'fully connected layers' , that connected all This results in many connections, and thus many weights to be learned, in the network. Note that our input dimension is now quite high (even with small pictures of `64x64` pixels): we have 12288 features. +(parameters-exercise-1)= :::: challenge -## Number of parameters{#parameters-exercise-1} + +## Number of parameters + Suppose we create a single Dense (fully connected) layer with 100 hidden units that connect to the input pixels, how many parameters does this layer have? - A. 1228800 @@ -265,9 +304,12 @@ in the context of applying a _Gaussian blur_. ::: :::: +(parameters-exercise-3)= :::: challenge -## Number of model parameters{#parameters-exercise-3} -Suppose we apply a convolutional layer with 100 kernels of size 3 * 3 * 3 (the last dimension applies to the rgb channels) to our images of 64 * 64 * 3 pixels. How many parameters do we have? Assume, for simplicity, that the kernels do not use bias terms. Compare this to the answer of the earlier exercise, ["Number of Parameters"](#parameters-exercise-1). + +## Number of model parameters + +Suppose we apply a convolutional layer with 100 kernels of size 3 _3_ 3 (the last dimension applies to the rgb channels) to our images of 64 _64_ 3 pixels. How many parameters do we have? Assume, for simplicity, that the kernels do not use bias terms. Compare this to the answer of the earlier exercise, ["Number of Parameters"](#parameters-exercise-1). ::: solution ## Solution @@ -275,7 +317,13 @@ We have 100 matrices with 3 * 3 * 3 = 27 values each so that gives 27 * 100 = 27 ::: :::: -So let us look at a network with a few convolutional layers. We need to finish with a Dense layer to connect the output cells of the convolutional layer to the outputs for our classes. +So let us look at a network with a few convolutional layers. + +::::::: group-tab + +###### Keras + +We need to finish with a fully-connected `Dense` layer to connect the output cells of the convolutional layer to the outputs for our classes. ```python from tensorflow import keras @@ -316,6 +364,59 @@ Model: "dollar_street_model_small" Non-trainable params: 0 (0.00 B) ``` + + +###### PyTorch + +We need to finish with a fully-connected `Linear` layer to connect the output cells of the convolutional layer to the outputs for our classes. + +```python +class DollarStreetModelSmall(nn.Module): + def __init__(self): + super().__init__() + self.s = nn.Sequential( + nn.Conv2d(3, 50, 3), + nn.ReLU(), + nn.Conv2d(50, 50, 3), + nn.ReLU(), + nn.Flatten(), + nn.Linear(50*60*60, 10), + ) + def forward(self, x): + return self.s(x) + +model = DollarStreetModelSmall() +summary(model, input_size=(1, 3, 64, 64)) +``` + +```output +========================================================================================== +Layer (type:depth-idx) Output Shape Param # +========================================================================================== +DollarStreetModelSmall [1, 10] -- +├─Sequential: 1-1 [1, 10] -- +│ └─Conv2d: 2-1 [1, 50, 62, 62] 1,400 +│ └─ReLU: 2-2 [1, 50, 62, 62] -- +│ └─Conv2d: 2-3 [1, 50, 60, 60] 22,550 +│ └─ReLU: 2-4 [1, 50, 60, 60] -- +│ └─Flatten: 2-5 [1, 180000] -- +│ └─Linear: 2-6 [1, 10] 1,800,010 +========================================================================================== +Total params: 1,823,960 +Trainable params: 1,823,960 +Non-trainable params: 0 +Total mult-adds (Units.MEGABYTES): 88.36 +========================================================================================== +Input size (MB): 0.05 +Forward/backward pass size (MB): 2.98 +Params size (MB): 7.30 +Estimated Total Size (MB): 10.32 +========================================================================================== +``` + + +::::::: + :::: challenge ## Convolutional Neural Network @@ -329,7 +430,7 @@ We can get inspiration for neural network architectures that could work on our d ::: solution ## Solution * The Flatten layer converts the 60x60x50 output of the convolutional layer into a single one-dimensional vector, that can be used as input for a dense layer. -* The last dense layer has the most parameters. This layer connects every single output 'pixel' from the convolutional layer to the 10 output classes. +* The last fully-connected `Dense` / `Linear` layer has the most parameters. This layer connects every single output 'pixel' from the convolutional layer to the 10 output classes. That results in a large number of connections, so a large number of parameters. This undermines a bit the expressiveness of the convolutional layers, that have much fewer parameters. ::: :::: @@ -385,6 +486,9 @@ Often in convolutional neural networks, the convolutional layers are intertwined Let's put it into practice. We compose a Convolutional network with two convolutional layers and two pooling layers. +::::::: group-tab + +###### Keras ```python def create_nn(input_shape): @@ -434,12 +538,74 @@ Model: "dollar_street_model" Non-trainable params: 0 (0.00 B) ``` + + +###### PyTorch + +```python +class DollarStreetModel(nn.Module): + def __init__(self): + super().__init__() + self.s = nn.Sequential( + nn.Conv2d(3, 50, 3), + nn.ReLU(), + nn.MaxPool2d(2), + nn.Conv2d(50, 50, 3), + nn.ReLU(), + nn.MaxPool2d(2), + nn.Flatten(), + nn.Linear(14*14*50, 50), + nn.ReLU(), + nn.Linear(50, 10), + ) + def forward(self, x): + return self.s(x) + +model = DollarStreetModel() +summary(model, input_size=(1, 3, 64, 64)) +``` + +```output +========================================================================================== +Layer (type:depth-idx) Output Shape Param # +========================================================================================== +DollarStreetModel [1, 10] -- +├─Sequential: 1-1 [1, 10] -- +│ └─Conv2d: 2-1 [1, 50, 62, 62] 1,400 +│ └─ReLU: 2-2 [1, 50, 62, 62] -- +│ └─MaxPool2d: 2-3 [1, 50, 31, 31] -- +│ └─Conv2d: 2-4 [1, 50, 29, 29] 22,550 +│ └─ReLU: 2-5 [1, 50, 29, 29] -- +│ └─MaxPool2d: 2-6 [1, 50, 14, 14] -- +│ └─Flatten: 2-7 [1, 9800] -- +│ └─Linear: 2-8 [1, 50] 490,050 +│ └─ReLU: 2-9 [1, 50] -- +│ └─Linear: 2-10 [1, 10] 510 +========================================================================================== +Total params: 514,510 +Trainable params: 514,510 +Non-trainable params: 0 +Total mult-adds (Units.MEGABYTES): 24.84 +========================================================================================== +Input size (MB): 0.05 +Forward/backward pass size (MB): 1.87 +Params size (MB): 2.06 +Estimated Total Size (MB): 3.98 +========================================================================================== +``` + + +::::::: + ## 5. Choose a loss function and optimizer We compile the model using the adam optimizer (other optimizers could also be used here!). Similar to the penguin classification task, we will use the crossentropy function to calculate the model's loss. This loss function is appropriate to use when the data has two or more label classes. +::::::: group-tab + +###### Keras Remember that our target class is represented by a single integer, whereas the output of our network has 10 nodes, one for each class. So, we should have actually one-hot encoded the targets and used a softmax activation for the neurons in our output layer! Luckily, there is a quick fix to calculate crossentropy loss for data that @@ -447,7 +613,6 @@ has its classes represented by integers, the `SparseCategoricalCrossentropy()` f Adding the argument `from_logits=True` accounts for the fact that the output has a linear activation instead of softmax. This is what is often done in practice, because it spares you from having to worry about one-hot encoding. - ```python def compile_model(model): model.compile(optimizer='adam', @@ -456,6 +621,20 @@ def compile_model(model): compile_model(model) ``` + + +###### PyTorch + +We choose the optimizer and loss function classes and will instantiate them inside the `fit` function. + +```python +optim = torch.optim.Adam +loss_fn = torch.nn.CrossEntropyLoss +``` + + +::::::: + :::callout ## Choosing a Metric @@ -469,7 +648,7 @@ If the data is highly imbalanced and 90 of these images are dogs, the model will This high number looks like the model performs great, but it is misleading; the model might not have learned to identify a cat image at all. In such an imbalanced dataset, other metrics such as [precision](https://keras.io/api/metrics/classification_metrics/#precision-class) and [recall](https://keras.io/api/metrics/classification_metrics/#recall-class) are more suitable. -The documentation provides a comprehensive list of [metrics available in Keras](https://keras.io/api/metrics/), suitable for different tasks and datasets. +The documentation provides a comprehensive list of [metrics available in Keras](https://keras.io/api/metrics/), suitable for different tasks and datasets. Similarly, the [`torchmetrics` package provides a variety of metrics for deep learning in PyTorch](https://lightning.ai/docs/torchmetrics/stable/all-metrics.html). ::: ::: instructor @@ -479,13 +658,79 @@ This is a good time for switching instructor and/or a break. ## 6. Train the model -We then train the model for 10 epochs: +We then train the model: + +::::::: group-tab + +###### Keras ```python history = model.fit(train_images, train_labels, epochs=10, validation_data=(val_images, val_labels)) ``` + + +###### PyTorch + +Here we define a `fit` function which iterates over several epochs and trains the model while computing and recording the history of losses and metrics, just as it was done in the previous episode. +The `train` and `test` functions are defined to iterate over the training and validation dataloaders for one epoch each. +Refer to the notebook to see how they are defined exactly. + +```python +def train(dl, model, loss_fn, optimizer, device=torch.device("cuda:0")): + model = model.to(device) + model.train() + + avg_loss = 0.0 + avg_acc = 0.0 + + for step, (x, y) in enumerate(tqdm(dl)): + # Train + # Compute losses and metric (accuracy) + ... + + return avg_loss, avg_acc + + +def test(dl, model, loss_fn, device=torch.device("cuda:0")): + model = model.to(device) + model.eval() + ... + + with torch.no_grad(): + for step, (x, y) in enumerate(dl): + # Compute losses and metric (accuracy) + ... + + return avg_loss, avg_acc + + +def fit(model, optimizer, loss, train_ds, val_ds, batch_size=32, learning_rate=0.001, num_epochs=20): + + # instantiate optimizer and loss_fn + loss_fn = loss() + optim = optimizer(model.parameters(), lr=learning_rate) + + train_dl = torch.utils.data.DataLoader(train_ds, batch_size=batch_size, shuffle=True) + val_dl = torch.utils.data.DataLoader(val_ds, batch_size=1, shuffle=False) + + history = {"loss": [], "val_loss": [], "accuracy": [], "val_accuracy": []} + for epoch in range(num_epochs): + train_loss, train_acc = train(train_dl, model, loss_fn, optim, device=device) + val_loss, val_acc = test(val_dl, model, loss_fn, device=device) + + for k, v in [("loss", train_loss), ("val_loss", val_loss), ("accuracy", train_acc), ("val_accuracy", val_acc)]: + history[k].append(v) + + return history + +history = fit(model, optim, loss_fn, train_ds, val_ds, learning_rate=0.0001) +``` + + +::::::: + ## 7. Perform a Prediction/Classification Here we skip performing a prediction, and continue to measuring the performance. In practice, you will only do this step once in a while when you actually need to have the individual predictions, @@ -538,11 +783,16 @@ This will take 30-45 minutes and might deviate the focus away from CNNs. 3. You can just mention that a simple network with only dense layers reaches 18% accuracy, considerably worse than our simple CNN. ::: -::: callout +::::::::::: callout ## Comparison with a network with only dense layers How does this simple CNN compare to a neural network with only dense layers? We can define a neural network with only dense layers: + +::::::: group-tab + +###### Keras + ```python def create_dense_model(): inputs = keras.Input(shape=train_images.shape[1:]) @@ -579,30 +829,81 @@ Model: "dense_model" Non-trainable params: 0 (0.00 B) ``` -As you can see this model has more parameters than our simple CNN, let's train and evaluate it! + + + +###### PyTorch ```python -compile_model(dense_model) -history = dense_model.fit(train_images, train_labels, epochs=20, - validation_data=(val_images, val_labels)) -plot_history(history, ['accuracy', 'val_accuracy']) +class DenseModel(nn.Module): + def __init__(self): + super().__init__() + self.s = nn.Sequential( + nn.Flatten(), + nn.Linear(12288, 50), + nn.ReLU(), + nn.Linear(50, 50), + nn.ReLU(), + nn.Linear(50, 10), + ) + def forward(self, x): + return self.s(x) + +dense_model = DenseModel() +summary(dense_model, input_size=(1, 3, 64, 64)) +``` + +```output +========================================================================================== +Layer (type:depth-idx) Output Shape Param # +========================================================================================== +DenseModel [1, 10] -- +├─Sequential: 1-1 [1, 10] -- +│ └─Flatten: 2-1 [1, 12288] -- +│ └─Linear: 2-2 [1, 50] 614,450 +│ └─ReLU: 2-3 [1, 50] -- +│ └─Linear: 2-4 [1, 50] 2,550 +│ └─ReLU: 2-5 [1, 50] -- +│ └─Linear: 2-6 [1, 10] 510 +========================================================================================== +Total params: 617,510 +Trainable params: 617,510 +Non-trainable params: 0 +Total mult-adds (Units.MEGABYTES): 0.62 +========================================================================================== +Input size (MB): 0.05 +Forward/backward pass size (MB): 0.00 +Params size (MB): 2.47 +Estimated Total Size (MB): 2.52 +========================================================================================== ``` + + +::::::: + +As you can see this model has more parameters than our simple CNN, let's train and evaluate it! + ![](fig/04_dense_model_training_history.png){alt="Plot of training accuracy and validation accuracy vs epochs for a model with only dense layers, showing training accuracy increasing to approximately 0.22 and validation accuracy plateauing around 0.18. Both values show relatively large fluctations as training progresses."} As you can see the validation accuracy only reaches about 18%, whereas the CNN reached about 28% accuracy. This demonstrates that convolutional layers are a big improvement over dense layers for these kind of datasets. -::: +::::::::::: ## 9. Refine the model -:::: challenge +::::::::::: challenge ## Network depth What, do you think, will be the effect of adding a convolutional layer to your model? Will this model have more or fewer parameters? Try it out. Create a `model` that has an additional `Conv2d` layer with 50 filters and another MaxPooling2D layer after the last MaxPooling2D layer. Train it for 10 epochs and plot the results. **HINT**: The model definition that we used previously needs to be adjusted as follows: + +::::::: group-tab + +###### Keras + ```python inputs = keras.Input(shape=train_images.shape[1:]) x = keras.layers.Conv2D(50, (3, 3), activation='relu')(inputs) @@ -615,10 +916,43 @@ x = keras.layers.Dense(50, activation='relu')(x) outputs = keras.layers.Dense(10)(x) ``` -::: solution + + +###### PyTorch + +```python +class DollarStreetModel(nn.Module): + def __init__(self): + super().__init__() + self.s = nn.Sequential( + nn.Conv2d(3, 50, 3), + nn.ReLU(), + nn.MaxPool2d(2), + nn.Conv2d(50, 50, 3), + nn.ReLU(), + nn.MaxPool2d(2), + # Add your extra layers here + nn.Flatten(), + nn.Linear(14*14*50, 50), + nn.ReLU(), + nn.Linear(50, 10), + ) + def forward(self, x): + return self.s(x) +``` + + +::::::: + +::::::::: solution ## Solution We add an extra Conv2D layer after the second pooling layer: + +::::::: group-tab + +###### Keras + ```python def create_nn_extra_layer(): inputs = keras.Input(shape=train_images.shape[1:]) @@ -691,8 +1025,38 @@ plot_history(history, ['accuracy', 'val_accuracy']) ![](fig/04_training_history_2.png){alt="Plot of training accuracy and validation accuracy vs epochs for the trained model, showing training accuracy increasing steadily by approximately 0.04 per epoch up to around 0.55 while validation accuracy increases before plateauing around 0.25."} -::: -:::: + + +###### PyTorch + +```py +class DollarStreetModel(nn.Module): + def __init__(self): + super().__init__() + self.s = nn.Sequential( + nn.Conv2d(3, 50, 3), + nn.ReLU(), + nn.MaxPool2d(2), + nn.Conv2d(50, 50, 3), + nn.ReLU(), + nn.MaxPool2d(2), + nn.Conv2d(50, 50, 3), + nn.ReLU(), + nn.MaxPool2d(2), + nn.Flatten(), + nn.Linear(6*6*50, 50), + nn.ReLU(), + nn.Linear(50, 10), + ) + def forward(self, x): + return self.s(x) +``` + + +::::::: + +::::::::: +::::::::::: ::: callout ## Other types of data @@ -747,6 +1111,10 @@ In practice, however, dropout is computationally a very elegant solution which d Let us add a dropout layer after each pooling layer towards the end of the network that randomly drops 80% of the nodes. +::::::: group-tab + +###### Keras + ```python def create_nn_with_dropout(): inputs = keras.Input(shape=train_images.shape[1:]) @@ -814,21 +1182,81 @@ Model: "dropout_model" Non-trainable params: 0 (0.00 B) ``` -We can see that the dropout does not alter the dimensions of the image, and has zero parameters. + -We again compile and train the model. -```python -compile_model(model_dropout) +###### PyTorch -history = model_dropout.fit(train_images, train_labels, epochs=20, - validation_data=(val_images, val_labels)) +```python +class ModelDropout(nn.Module): + def __init__(self): + super().__init__() + self.s = nn.Sequential( + nn.Conv2d(3, 50, 3), + nn.ReLU(), + nn.MaxPool2d(2), + nn.Dropout(0.5), # This is new! + nn.Conv2d(50, 50, 3), + nn.ReLU(), + nn.MaxPool2d(2), + nn.Dropout(0.5), # This is new! + nn.Conv2d(50, 50, 3), + nn.ReLU(), + nn.MaxPool2d(2), + nn.Dropout(0.5), # This is new! + nn.Flatten(), + nn.Linear(1800, 50), + nn.ReLU(), + nn.Linear(50, 10), + ) + def forward(self, x): + return self.s(x) + +dropout_model = ModelDropout() +summary(dropout_model, input_size=(1, 3, 64, 64)) ``` -And inspect the training results: -```python -plot_history(history, ['accuracy', 'val_accuracy']) +```output +========================================================================================== +Layer (type:depth-idx) Output Shape Param # +========================================================================================== +ModelDropout [1, 10] -- +├─Sequential: 1-1 [1, 10] -- +│ └─Conv2d: 2-1 [1, 50, 62, 62] 1,400 +│ └─ReLU: 2-2 [1, 50, 62, 62] -- +│ └─MaxPool2d: 2-3 [1, 50, 31, 31] -- +│ └─Dropout: 2-4 [1, 50, 31, 31] -- +│ └─Conv2d: 2-5 [1, 50, 29, 29] 22,550 +│ └─ReLU: 2-6 [1, 50, 29, 29] -- +│ └─MaxPool2d: 2-7 [1, 50, 14, 14] -- +│ └─Dropout: 2-8 [1, 50, 14, 14] -- +│ └─Conv2d: 2-9 [1, 50, 12, 12] 22,550 +│ └─ReLU: 2-10 [1, 50, 12, 12] -- +│ └─MaxPool2d: 2-11 [1, 50, 6, 6] -- +│ └─Dropout: 2-12 [1, 50, 6, 6] -- +│ └─Flatten: 2-13 [1, 1800] -- +│ └─Linear: 2-14 [1, 50] 90,050 +│ └─ReLU: 2-15 [1, 50] -- +│ └─Linear: 2-16 [1, 10] 510 +========================================================================================== +Total params: 137,060 +Trainable params: 137,060 +Non-trainable params: 0 +Total mult-adds (Units.MEGABYTES): 27.68 +========================================================================================== +Input size (MB): 0.05 +Forward/backward pass size (MB): 1.93 +Params size (MB): 0.55 +Estimated Total Size (MB): 2.53 +========================================================================================== ``` + +::::::: + +We can see that the dropout does not alter the dimensions of the image, and has zero parameters. + +We again compile and train the model, and inspect the training results. + ![](fig/04_training_history_3.png){alt="Plot of training accuracy and validation accuracy vs epochs for the trained model, showing both values increasing before they diverge after around 10 epochs, with training accuracy reaching approximately 0.4 while validation accuracy plateaus around 0.3"} Now we see that the gap between the training accuracy and validation accuracy is much smaller, and that the final accuracy on the validation set is higher than without dropout. @@ -905,7 +1333,11 @@ The goal is to show that hyperparameter tuning can be done easily with `keras_tu ::: Recall that hyperparameters are model configuration settings that are chosen before the training process and affect the model's learning behavior and performance, for example the dropout rate. In general, if you are varying hyperparameters to find the combination of hyperparameters with the best model performance this is called hyperparameter tuning. A naive way to do this is to write a for-loop and train a slightly different model in every cycle. -However, it is better to use the `keras_tuner` package for this. +However, it is better to use existing packages for this. + +::::::: group-tab + +###### Keras Let's first define a function that creates a neuronal network given 2 hyperparameters, namely the dropout rate and the number of layers: ```python @@ -923,10 +1355,41 @@ def create_nn_with_hp(dropout_rate, n_layers): return model ``` + + +###### PyTorch + +```python +class HPModel(nn.Module): + def __init__(self, dropout_rate, n_layers): + super().__init__() + self.in_layer = nn.Sequential(nn.Conv2d(3, 50, 3), nn.ReLU(), nn.MaxPool2d(2)) + self.hidden_layers = nn.Sequential(*[nn.Sequential(nn.Conv2d(50, 50, 3), nn.ReLU(), nn.MaxPool2d(2)) for n in range(n_layers-1)]) + n_features_after_flatten = 64 + for n in range(n_layers): + n_features_after_flatten = (n_features_after_flatten - 2)//2 + + n_features_after_flatten = n_features_after_flatten ** 2 * 50 + self.head = nn.Sequential(nn.Dropout(dropout_rate), nn.Flatten(), nn.Linear(n_features_after_flatten , 50), nn.ReLU(), nn.Linear(50, 10)) + + def forward(self, x): + x = self.in_layer(x) + x = self.hidden_layers(x) + x = self.head(x) + return x +``` + + +::::::: + Now, let's find the best combination of hyperparameters using grid search. Grid search is the simplest hyperparameter tuning strategy, you test all the combinations of predefined values for the hyperparameters that you want to vary. +::::::: group-tab + +###### Keras + For this we will make use of the package `keras_tuner`, we can install it by typing in the command line: ```bash pip install keras_tuner @@ -1008,6 +1471,51 @@ dropout_rate: 0.5 Score: 2.143627882003784 ``` + + +###### PyTorch + +```python +n_layers_grid = [1, 2] +dropout_rate_grid = np.linspace(.2, .8, 3) + +best = None +best_loss = float("inf") +best_model = None +histories = {} +trial = 0 + +for n_layers in n_layers_grid: + for dropout_rate in dropout_rate_grid: + + history = fit(HPModel(dropout_rate, n_layers), optim, loss_fn, train_ds, val_ds, learning_rate=0.0001) + histories[n_layers, dropout_rate] = history + + val_loss = min(history["loss"]) + if val_loss < best_loss: + best_loss = val_loss + best = (n_layers, float(dropout_rate),) + best_model = model + + print( + f"trial {trial}: params: {n_layers=}, {dropout_rate}; " + f"best: {best_loss} at (n_layers, dropout_rate)={best}" + ) + trial += 1 + +print(f"best val loss: {best_loss:.4f} at (n_layers, dropout_rate)={best}") +print("trials executed:", trial) +``` + +```output +[...more output here...] +best val loss: 1.5804577001503535 at (n_layers, dropout_rate)={(1, 0.8) +trials executed: 6 +``` + + +::::::: + :::: challenge ## Hyperparameter tuning @@ -1119,11 +1627,7 @@ Instead, `random search` randomly samples combinations of hyperparemeters, allow Next to grid search and random search there are many different hyperparameter tuning strategies, including [neural architecture search](https://en.wikipedia.org/wiki/Neural_architecture_search) where a separate neural network is trained to find the best architecture for a model! ## 10. Share model -Let's save our model - -```python -model.save('cnn_model.keras') -``` +Let's save our model as done previously. ## Conclusion and next steps How successful were we with creating a model here? diff --git a/episodes/5-transfer-learning.md b/episodes/5-transfer-learning.md index d8978f726..aa49d3ff5 100644 --- a/episodes/5-transfer-learning.md +++ b/episodes/5-transfer-learning.md @@ -30,7 +30,7 @@ flowchart LR C --> E(Dog Breeds Model) ``` -In this episode we will learn how use Keras to adapt a state-of-the-art pre-trained model to the [Dollar Street Dataset](https://zenodo.org/records/10970014). +In this episode we will learn how to adapt a state-of-the-art pre-trained model to the [Dollar Street Dataset](https://zenodo.org/records/10970014). ## 1. Formulate / Outline the problem @@ -56,19 +56,106 @@ The goal is to predict one out of 10 classes to which the image belongs. ## 3. Prepare the data + +### Import the deep learning framework + +::::::: group-tab + +###### Keras + +Before we move on to the next section of the workflow we need to make sure we have Keras imported. +We do this as follows: + +```python +from tensorflow import keras +import tensorflow as tf + +keras.utils.set_random_seed(2) + +``` + + + +###### PyTorch + +Before we move on to the next section of the workflow we need to make sure we have PyTorch imported. +We do this as follows: + +```python +import torch +import torchvision + +torch.manual_seed(2) +``` + + +::::::: + + +::::::: group-tab + +###### Keras We prepare the data as before, scaling the values between 0 and 1. + ```python train_images = train_images / 255.0 val_images = val_images / 255.0 ``` + + +###### PyTorch + +We introduce a dataset class here: +- Similar as in the previous session: It scales the values between 0 and 1, but also permutes the order of dimensions as image data is organised differently in PyTorch than in tensorflow. +- What is new is that, **we introduce an callable attribute `transform` to the dataset class**. This function preprocesses the images before feeding them to our neural network. It is a good practice to do this in the dataset class as then the transformations are only executed for an image when fetching the image. In our lesson here it would be also fine to do these once, but in many cases one would then run out of memory. + +```python +class DollarStreetDataset(torch.utils.data.Dataset): + def __init__(self, root, train=True, transform = None): + prefix = "test" + if train == True: + prefix = "train" + self.images = np.load(root / f'{prefix}_images.npy') / 255. + self.labels = np.load(root / f'{prefix}_labels.npy') + self.transform = transform + + def __getitem__(self, idx): + # PyTorch requires another order of dimensions than the original tensorflow data + x = torch.permute(torch.tensor(self.images[idx], dtype=torch.float), (2, 0, 1)) + + # This is required as we would like to perform the transforms when fetching the data + # Especially important, e.g., with large datasets or when doing augmentations + if self.transform is not None: + x = self.transform(x) + y = torch.tensor(self.labels[idx], dtype=torch.long) + return x, y + + def __len__(self): + return self.images.shape[0] +``` + + + +::::::: + + ## 4. Choose a pre-trained model or start building architecture from scratch + +::: note +In practice, you would probably first pick a pre-trained model, and then realise what pre-processing needs to be done. The steps below can be seen as the results of an iterative work. +::: + +Before loading any pre-trained model, we need to take care of the fact that our images have 64 x 64 pixels, whereas the pre-trained model that we will use was +trained on images of 160 x 160 pixels. + +::::::: group-tab + +###### Keras + Let's define our model input layer using the shape of our training images: ```python # input tensor -from tensorflow import keras -keras.utils.set_random_seed(2) - inputs = keras.Input(train_images.shape[1:]) ``` @@ -78,19 +165,51 @@ To adapt our data accordingly, we add an upscale layer that resizes the images t ```python # upscale layer -import tensorflow as tf method = tf.image.ResizeMethod.BILINEAR upscale = keras.layers.Lambda( lambda x: tf.image.resize_with_pad(x, 160, 160, method=method))(inputs) ``` + -From the `keras.applications` module we use the `DenseNet121` architecture. -This architecture was proposed by the paper: [Densely Connected Convolutional Networks (CVPR 2017)](https://arxiv.org/abs/1608.06993). It is trained on the [Imagenet](https://www.image-net.org/) dataset, which contains 14,197,122 annotated images according to the WordNet hierarchy with over 20,000 classes. +###### PyTorch + +Our images are 64 x 64 pixels, whereas the pre-trained model that we will use was +trained on images of 160 x 160 pixels. +To adapt our data accordingly, we define a `transform` function that resizes the images to 160 x 160 pixels during training and prediction. + +```python +import torchvision.transforms as T + +transform = T.Compose([ + T.ToPILImage(), + T.Resize(160), # keeps aspect ratio + T.CenterCrop(160), # ensures output is 160x160 + T.ToTensor()]) +``` + +We pass our `transform` function when creating the `train_dataset` and `val_dataset`. + +``` +train_dataset = DollarStreetDataset(DATA_FOLDER, train=True, transform=transform) +val_dataset = DollarStreetDataset(DATA_FOLDER, train=False, transform=transform) +``` + + +::::::: + +We use a DenseNet121. This architecture was proposed by the paper: [Densely Connected Convolutional Networks (CVPR 2017)](https://arxiv.org/abs/1608.06993). +It is trained on the [Imagenet](https://www.image-net.org/) dataset, which contains 14,197,122 annotated images according to the WordNet hierarchy with over 20,000 classes. We will have a look at the architecture later, for now it is enough to know that it is a convolutional neural network with 121 layers that was designed to work well on image classification tasks. +::::::: group-tab + +###### Keras +From the `keras.applications` module we use the `DenseNet121` architecture. + + Let's configure the DenseNet121: ```python base_model = keras.applications.DenseNet121(include_top=False, @@ -129,6 +248,53 @@ this network on the Imagenet data. We connect the network to the `upscale` layer that we defined before. + + +###### PyTorch +From the `torchvision.models` module we use the `densenet121` architecture. + + +```python +import torch.nn as nn + +# We would like to specify a sensible path for this as this is where the pre-trained model is stored. +os.environ['TORCH_HOME']='.' + +class DenseNetClassifier(nn.Module): + def __init__(self, num_classes): + super().__init__() + + # taking the pre-trained model from torchvision here, but throwing away the head + self.backbone = nn.Sequential( + torchvision.models.densenet121(weights="IMAGENET1K_V1").features, + nn.AdaptiveMaxPool2d((1, 1)), + nn.Flatten()) + + # creating our own head for classification + self.head = nn.Sequential(nn.BatchNorm1d(1024), + nn.Linear(1024, 50), + nn.ReLU(), + nn.Dropout(0.5), + nn.Linear(50, num_classes)) + + def forward(self, x): + x = self.backbone(x) # → [batch_dim, 1024] + x = self.head(x) # → [batch_dim, num_classes] + return x +``` + +We would like to use only the feature extractor of the DenseNet and add our own classification head. +Thus, we use the `torchvision.models.densenet121(weights="IMAGENET1K_V1").features` subpart of the pre-trained DenseNet. +It maps each image to a vector of dimension `1024`. The `weights` parameter here specifies that we use ImageNet as pre-training data. +*Note that there are many different versions of ImageNet, but for this course it is fine that we use one version of ImageNet.* + +The `self.head` maps from the vector of dimension `1024` to a vector of dimension `num_classes` (10 in our case). +The choices here are arbitrary, one might as well as head only one layer of `nn.Linear(1024, num_classes)`. + + + +::::::: + ### Only train a 'head' network Instead of fine-tuning all the weights of the DenseNet121 network using our dataset, we choose to freeze all these weights and only train a so-called 'head network' @@ -136,6 +302,11 @@ that sits on top of the pre-trained network. You can see the DenseNet121 network as extracting a meaningful feature representation from our image. The head network will then be trained to decide on which of the 10 Dollar Street dataset classes the image belongs. +::::::: group-tab + +###### Keras + + We will turn of the `trainable` property of the base model: ```python base_model.trainable = False @@ -155,9 +326,65 @@ Finally we define our model: ```python model = keras.models.Model(inputs=inputs, outputs=out) ``` -:::: challenge + + + +###### PyTorch + +In PyTorch, we can prevent updating layers of the model by setting the `param.requires_grad` property to `False`. +Here, we do that for the `backbone` part of the model as we would like to train the `head`. + +```python +model = DenseNetClassifier(num_classes=10) + +for param in model.backbone.parameters(): + param.requires_grad = False +``` + +If you want to make sure, you could print all parameters of `backbone` and `head` and see if they are frozen or not. + +```python +print("Backbone parameters (should not be trainable)") +for name, param in list(model.backbone.named_parameters(prefix="backbone")): + print(name, "trainable:", param.requires_grad) + +print("\nBackbone parameters (should be trainable)") +for name, param in list(model.head.named_parameters(prefix="head")): + print(name, "trainable:", param.requires_grad) + +``` + + +::::::: + +:::::::::::: challenge ## Inspect the DenseNet121 network -Have a look at the network architecture with `model.summary()`. +Have a look at the network architecture: + +::::::: group-tab + +###### Keras + +Use the following function: + +```python +model.summary() +``` + + + +###### PyTorch +Use the following lines to generate the summary: + +```python +from torchinfo import summary +summary(model, depth=2) +``` +*(We set the `depth=2` to reduce the amount of lines printed.)* + + +::::::: + It is indeed a deep network, so expect a long summary! ### 1.Trainable parameters @@ -168,29 +395,56 @@ Why is this and how does it effect the time it takes to train the model? ### 2. Head and base Can you see in the model summary which part is the base network and which part is the head network? -### 3. Max pooling +### 3. Max pooling (relevant only for Keras) Which layer is added because we provided `pooling='max'` as argument for `DenseNet121()`? +:::::::::::: -::: solution +:::::::::::: solution ## Solutions +::::::: group-tab + +###### Keras + ### 1. Trainable parameters -Total number of parameters: 7093360, out of which only 53808 are trainable. -The 53808 trainable parameters are the weights of the head network. All other parameters are 'frozen' because we set `base_model.trainable=False`. Because only a small proportion of the parameters have to be updated at each training step, this will greatly speed up training time. +Total number of parameters: 7,093,360, out of which only 53,808 are trainable. + +The 53,808 trainable parameters are the weights of the head network. All other parameters are 'frozen' because we set `base_model.trainable=False`. Because only a small proportion of the parameters have to be updated at each training step, this will greatly speed up training time. ### 2. Head and base The head network starts at the `flatten` layer, 5 layers before the final layer. -### 3. Max pooling +### 3. Max pooling (relevant only for Keras) The `max_pool` layer right before the `flatten` layer is added because we provided `pooling='max'`. -::: -:::: + + + +###### PyTorch + +### 1. Trainable parameters + +Total number of parameters: 7,007,664, out of which only 53,808 are trainable. + +The 53,808 trainable parameters are the weights of the head network. All other parameters are 'frozen' because we set `param.requires_grad = False`. Because only a small proportion of the parameters have to be updated at each training step, this will greatly speed up training time. + +### 2. Head and base +The head network starts at the `flatten` layer, 5 layers before the final layer. + +::::::: -:::: challenge +:::::::::::: + + +:::::::::::::::::::::::::::: challenge ## Training and evaluating the pre-trained model +Note that we have added more hints for PyTorch here as it requires more lines of code, but allows also for more customization. + +::::::: group-tab + +###### Keras ### 1. Compile the model Compile the model: @@ -211,12 +465,151 @@ Train the model on the training dataset: Plot the training history and evaluate the trained model. What do you think of the results? ### 4. (Optional) Try out other pre-trained neural networks -Train and evaluate another pre-trained model from https://keras.io/api/applications/. How does it compare to DenseNet121? +Train and evaluate another pre-trained model from [keras applications](https://keras.io/api/applications/). How does it compare to DenseNet121? + + +###### PyTorch -::: solution +### 1. Define training and test loops + +We first define two functions similar to session 4 that `train` and `test` the model. + +```python +from tqdm import tqdm + +def train(dataloader, model, loss_fn, optimizer, device): + model.train() + train_loss = 0 + correct = 0 + total = 0 + + # this tdqm magic is only there to make this progress bar appear + for step, (x, y) in (enumerate(pbar:=tqdm(dataloader))): + x, y = x.to(device), y.to(device) + optimizer.zero_grad() + logits = model(x) + loss = loss_fn(logits, y) + loss.backward() + optimizer.step() + train_loss += loss.item() * x.size(0) + preds = logits.argmax(dim=1) + correct += (preds == y).sum().item() + total += y.size(0) + pbar.set_description(f"Loss: {(loss.item()):>7f}") + + train_loss /= total + train_acc = correct / total + + + return train_loss, train_acc + +def test(dataloder, model, loss_fn, device=torch.device("cuda:0")): + model.eval() + val_loss = 0 + correct = 0 + total = 0 + with torch.no_grad(): + for x, y in dataloder: + x, y = x.to(device), y.to(device) + logits = model(x) + loss = loss_fn(logits, y) + val_loss += loss.item() * x.size(0) + preds = logits.argmax(dim=1) + correct += (preds == y).sum().item() + total += y.size(0) + + val_loss /= total + val_acc = correct / total + + return val_loss, val_acc + +``` + +### 2. Train the model + +We use our `train` and `test` functions together with other objects know from prior sessions. + +Note that in PyTorch early stopping needs to be defined by you as there is no predefined class as in Keras. + +```python +from torch.utils.data import DataLoader +from torch.optim import Adam + + +# take a GPU as default when it is available +device = "cuda" if torch.cuda.is_available() else "cpu" + + +model = model.to(device) +optimizer = Adam(model.parameters(), lr=0.001) # learning rate is here set to the same as in Keras default +loss_fn = nn.CrossEntropyLoss() + +train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True) +val_loader = DataLoader(val_dataset, batch_size=32) + +num_epochs = 30 + +# for early stopping +best_val_loss = float("inf") +patience = 5 +wait = 0 + +# for bookkeeping +history = {"epoch": [], "train_loss": [], "train_acc": [], "val_loss": [], "val_acc": []} + +for epoch in range(num_epochs): + train_loss, train_acc = train(dataloader=train_loader, model=model, loss_fn=loss_fn, optimizer=optimizer, device=device) + val_loss, val_acc = test(dataloder=val_loader, model=model, loss_fn=loss_fn, device=device) + + + # Save to history + history["epoch"].append(str(epoch+1)) + history["train_loss"].append(train_loss) + history["train_acc"].append(train_acc) + history["val_loss"].append(val_loss) + history["val_acc"].append(val_acc) + + + # print progress + print(f"Epoch {epoch+1}/{num_epochs} ", + f"Train Loss: {train_loss:.4f} Train Acc: {train_acc:.4f} ", + f"Val Loss: {val_loss:.4f} Val Acc: {val_acc:.4f}") + + # Early stopping (there is not a build-in version for that in PyTorch) + if val_loss < best_val_loss: + best_val_loss = val_loss + wait = 0 + torch.save(model.state_dict(), "best_model.pt") + else: + wait += 1 + if wait >= patience: + print("Early stopping triggered.") + break + +print(f"Training complete at epoch {epoch+1}") + +``` + +### 3. Inspect the results + +You can plot the results yourself or have a look at the plotting in the solution. + +### 4. (Optional) Try out other pre-trained neural networks +Train and evaluate another pre-trained model from [torch vision](https://docs.pytorch.org/vision/main/models.html). How does it compare to DenseNet121? + + +::::::: + +:::::::::::::::::::::::::::: + +:::::::::::::::::::::::::::: solution ## Solution +::::::: group-tab + +###### Keras + ### 1. Compile the model ```python model.compile(optimizer='adam', @@ -261,8 +654,36 @@ plot_history(history, ['accuracy', 'val_accuracy']) ![](fig/05_training_history_transfer_learning.png){alt='Training history for training the pre-trained-model. The training accuracy slowly raises from 0.2 to 0.9 in 20 epochs. The validation accuracy starts higher at 0.25, but reaches a plateau around 0.64'} The final validation accuracy reaches 64%, this is a huge improvement over 30% accuracy we reached with the simple convolutional neural network that we build from scratch in the previous episode. -::: -:::: + + +###### PyTorch + +### 1. See solution in exercise + +### 2. See solution in exercise + +### 3. Inspect the results + +```python +def plot_history(history, metrics): + history_df = pd.DataFrame.from_dict(history) + for metric in metrics: + sns.lineplot(data=history_df, x="epoch",y=metric, label=metric, marker="o") + plt.xlabel("epochs") + plt.ylabel("metric") + +plot_history(history, ['train_acc', 'val_acc']) +``` + +![](fig/05_training_history_transfer_learning_pytorch.png){alt='Training history for training the pre-trained-model. The training accuracy slowly raises from 0.84 to 0.9 in 14 epochs. The validation accuracy starts higher, but reaches a plateau around 0.64'} +The final validation accuracy reaches 64%, this is a huge improvement over the accuracy we reached with the simple convolutional neural network that we build from scratch in the previous episode. + + + + +::::::: + +:::::::::::::::::::::::::::: ## Concluding: The power of transfer learning In many domains, large networks are available that have been trained on vast amounts of data, such as in computer vision and natural language processing. Using transfer learning, you can benefit from the knowledge that was captured from another machine learning task. In many fields, transfer learning will outperform models trained from scratch, especially if your dataset is small or of poor quality. @@ -272,5 +693,6 @@ Transfer learning adapts a model to a specific dataset. This typically leads to ::: keypoints - Large pre-trained models capture generic knowledge about a domain -- Use the `keras.applications` module to easily use pre-trained models for your own datasets +- Use the `keras.applications` or `torchvision.models` module to easily use pre-trained models for your own datasets +- As usual with all options, there can be drawbacks to using pre-trained models. ::: diff --git a/episodes/6-outlook.md b/episodes/6-outlook.md index c14173946..17b14066e 100644 --- a/episodes/6-outlook.md +++ b/episodes/6-outlook.md @@ -174,7 +174,7 @@ But there is still so much to learn and do! Here are some suggestions for next steps you can take in your endeavor to become a deep learning expert: -* Learn more by going through a few of [the learning resources we have compiled for you](learners/reference.md#external-references) +* Learn more by going through a few of [the learning resources we have compiled for you](./reference.md#external-references) * Apply what you have learned to your own projects. Use the deep learning workflow to structure your work. Start as simple as possible, and incrementally increase the complexity of your approach. * Compete in a [Kaggle competition](https://www.kaggle.com/competitions) to practice what you have learned. diff --git a/episodes/fig/05_training_history_transfer_learning_pytorch.png b/episodes/fig/05_training_history_transfer_learning_pytorch.png new file mode 100644 index 0000000000000000000000000000000000000000..d30712042be5f92582093eca281c5d2202d0f06d GIT binary patch literal 26609 zcma&O1yogGw=Rq#prlB*f`D{)siaa8(w)+qZV+h^5b2ig?(U9_G}7JOwYl&1ocsO% zxZ{j*uVY~Cz1DhTzB8WrJS+H%d^F+~J~$2dAUf18U>D;C_xHeh{_e-UZ66( zdS2c$)O2}!yZTGofM$7P;m76aaV^%b!}|qykq?}g$@apF_A&wbghAjJC$@<$!AnX? z%9RhOJm8Pf%r|r3zojD6AynX>vftZ7;oqYBfBEoAaBeV>CmfTE|2ZLH??{@cdV`Zy z^kz&{)X`d`{AivM?PEm5?b#ZZ5CWD9gDUvbMy{_hctD=2AB_*|g!r^Wk0qbYlAdu{7a zhM&K`K6||2J#{b`zYr_B@7$c$-%2yz8Lf`#AIqYm0qAVmGVv?0xn&5U)!#}slc9vq z+t;?Xw%rURjfw};TvJ+xKPW_+CreznMqVL~$nwlPjDg>Xm-oS3xSFSKE!VnDD$4Xb zkhA4e4}?v)6NxxX&-qBX^5zLOe9dFm*4BRh{P|W*O)YfuhH);5*Y$2%DrnYvXrHnzR#vW(Fl_!11J z2qmI1*M5~t=8yRGtG$Q9`$xxe?ujE9dDmb(2LmY<7M5x)Ha51Ik@=Y7%4xt##qp># z%bz-X(|X6K!eF@6VJ@PEYw%Yd&u2? zR914{obUFmbcM{zcIV~gt>2h^|Ni}IEEU$-?0NeX17p2>fO&PlW|4-SebkVPDBzuj z2C=%jI=Q>tadB_2l!2jPRaiuj6o)bU$C6NM8aq2XFzc5B0u7_EM8lr2i0Ei>2M5l! zwl+6kOEL70-@ku>A_xpe#>oD;A@W3bj6 zake`JrAjlJuXptG^GkA9prlkR`7TX5sM{PaUT_hq^o3TiH~d2fPMxU{?c=+|LnJ&@ zp7YERlO6-^NKNJUYQIICaw7E#4x8r#^p_yKN2djK0h2|V?rD_R*n9;3+ybYuhpSmp z^9P1}{!W*h&N|b7at;NWGtKhcJ|$$L2sK6dc*A1e$+N6it>3<3&U3pwvA5kVswwxl zw$pLjrk0_>p@|IUS@gh(?CYG;^&V>VL45Y~sb1H2yuJB`Sa33b#l;c67ZoKEbYWfe zy0xm?yj`!1>N8q7^nMUf)%D~88{lv-7Y`MAXjG}w%Q7DDMFHb znZ^4WjZDBPQ#=Hpl;0s}fVtsEPO4YJ*RKc*Zo5p~VPpft$pU6m#oEy&Z1x*6Dc&$w z9hYS!9j|NS?r;idn=gvlOu2DY6<@O4-)gG?6bg^G1u8{qRpxjtxATWVAt9?byCo<~ z6RNt@e0=eysh%-lC{cTRmMXO>qsVv)?Rt+Sl+pOO`@J@+~>)n|s zxV~(8*vI7YhMlIm9<`y=9X8+2ZDNoM5`ML5xsR!`m@4#A4`2_~r-(4MROt9K42;wtqfsg6^{r#scxq%Nvsa1I)>~{7hLo&ShtSphoCfjPZ z`aqA0a>ZhT>llBUbWbE+_f;)vKjYdLr&nB7F1Td)pKO)0qIVZ3(;lr$z|gFmXfWg% zd6Yhxj~ng8cSnq z0++>Cj#+Butn6EHvBox5^m5dX5JF^|BH`A#D4~PfkD6FX0q+j4o_bwg1kk6__AZ`; zL`1y&TZKxKCJx^Z9HXN9!~2&76qi5to_sR=%^ELLiHY_bY%bz#P41CamlKMl9G=Px zFI2uL^|v}@$(2czqjulb+<&j+LKkTePX~(?o6;dcEYL-o7DD zuh1ysCFQzAF4C&|1vgfZ!w^G4*GPFGt*hrzo=wXgf%Z_L%i8e$`G%=`TQxO85UXEO zQ~N7qNPKqjNEPz4FK(S+vDS2%JELGEcsn!AF$mshz-LF$W&G;* zlpl7=eBBCt-!qx!U`eL2EZo7$`}vb1^I6n=ZEk1VkzA1!>~iB{s{5b(dl{l=;Mq;_-1tMPNjURJbrA1AG#zFK#&{uBQ^Qd7$?Ra(8u zOz2G|-_;HbZTQ+kh$L&d_%lerRj_oQTCvA}Au`A^4vKDm7J+xbV$0h{|0eI^_Lrve zN->lK*Lue%=hxXx9JduQ4y|0KWH>o4K7RR^Kj|N#i9+t&k+kwAS4Ybre>y(uz>18F zRI+iLcil|3{974pyV?yB(c0r~lDf?l_bcFg=Qm}Ncq7u&DPcELE$mi*In&eA;g&sr z>3#M`5^l-Bb4A?Ud4NpHM-fSE zXz5Hcx|^F>w2hjG42AY?fe2K5J36-|A=*x*suHzK<>?#Xw}LRS90KDe^&k16+)TLT z=a#ps?lk{NPTX&IUASsizJ%RN4;|8iDx>O`w%R@JVBXye_rz}+-+lAs7B^quI< zzZJC89IcOc03{6(}zqcsB@TMaJebR;P;d}tVuAp5dh|DAH4 zF9ApQ!_yVX``Uy6T!7Y4xsiy%yfld&jLJRw+H6fiEHjwCbSD0Oh#9|lE`01Oej7|s zxSCe=!eaGLa-KLbv47%?0Rut^1+KBE-V@$EJ&X&-&y=Gwh0~tjwo~sq(+1xo6z@z0 z5IszF6bNvHTz${Z+VO3Asxs_Dz~6!Pq1an>*p{pWrKAPM4~n_AZHE{Euk~nEcGnZ5 zi-Y-rXjPpLGBW6CX=&)>f^i_OR}Yu)xL;VhT^`6vNj-h`?AaV}wrlI_(E_IH{n0;2 z`F<@e>1kAcX`_zSYR^+DMwZAK>pO*t$SW$o098+=#Z(x`$?AvCgL;Y7k+{ikmL#op zN&`gW{VRi?8vQBqjkg!|Mj|oe#*HKpPgx%kQ<$TS0bcSqM?ctIwyPqOhi#OMTWOB)iW7&`T z!*4BBU(V(Yulnif{qY+$QS73b91)el{G`hX9|800Gr^L_$J8sHf?Z5)%`pcPEQn!5-Iw zr~$pHGS3)I&5**0iP7%-0j(}S+jKzql`iUfP}4*--JBK9fODL*v(V!Gfr=*uml*My zmL7|HiC+iO?wAr7N+FUGE_@RNYIFqHHPqX#1A|A8JF#zIzz`q?4u=cLpa2AMY^K@M zJz3Ja`TjyvBAi^-!$VMCUw_VSfGJ7P_1Cv=kCz5}WsR$C7Uckn5PcaUwa=G~{Get3 zGsKj|bQ`*Tr71JVQ_rCbg9!r5m>mD4h4Y{{_4K2a`kgiv=jYHlzpl3Zsy*l95>Lbv zuw~jEqq<&wOKrZ_ix03K-G>`hglyTw?$Ipi>j7QZdtiz0-o4Xu->;gstkw|m&O=1O zJo0|H^q%v)oEN-58>_UO`Gv=z0-3d{(XW_!iit_{=1p#YkE(NO(D*|04?*&47c`4Y z)5jw$ZB>N2Q)*1zwCNBfoXEUMCDE_D^m%E{^=_74W?RWQB?pI5HtD&08r4Hz38cHjfKFs$>(>0XMXonmNIebv1`NAfQ4a!{k z3gnLKzT)ty#tq`OXFnW}5>F%BN?>kp;pNW5{o+HJ)hxGl^R>z4F{q4Ak9#OMoc7c} zEy-armS4{IcJXrH$VVi9JV~9 zfVdKtoBM{`>q;MX6eyxsr9s4I@Pvqn$a1zibQTnpNzGmn+5c4)$Ijm0fZ7nuk7_o2 z9Rcz#c2dyuGs_DN^Y~{Gto2!gS}k1ZVc0YC_we*2S4KJ9vk$nk+8&N)tLZ(ekTaK7rI#@t|8 z>`DK8+ecT4L%!Bi)FHrXtA=9fM9cK;-qq3)t9XZ?r0YtIKCK|Df8;Z_tglztXdK?( zy+PfB1gD!Ucb1PU>DGT#CI zW&QBtXg+KeL2FVVMiWHc17~;oBmpxQN~av00_1t|{E13$iy$dq6>aZ6Fuizo`n~}? zSLh95d}CC3;={g^m8JoA+voe$ZDNvcq-0G)_V|SRFELWubaTq-#TKnn9luj`S0gl) z6|KBKsQ82iy?mfK4}f(vM-Ix)4COZ!->=^d(i9UthYC?eZasH2@qxl?-zbF->yve6 z9~V&~ZZaWC0JZC~pnR0Z91C`IU?e0jd7C`@N>2vD=$w3nLc_xzEge_Y4y=Tf4z2`XGQALlT@&`M`=e-a($JH;G|P5Hyqt%&<2H;D8L*SlL!OaDzUr_ zPSY5LYj;)e3@6$?-b&vwnyBmxwV~`{RTya!cR5W(Wu(q@_7?xq;>l&osFyH2w@`aG z2}0STYf`M@#nEaSBf{gm!8zVcGH31S;OVDK%yf3u!zS1w06RObSJ$=Lta>=;FyKDE zSbKG5K@hyia-ndu{#kLxpW5SiOhH)9J8mck5%F!oa|YFmkj?scR@>y8vky5K9eTi_ z58hkzXolBT^FBM{u|FNTxI6e3CKkvr9yIK#FzWqD z4^8DT#Vl#Qp$8x&s26PatESj3CJQxKDoP+!E1EWU&J^SZ!MAsI%6443Z~NqtZ*aVvzE6&5pWUHwYcbGWizaTN6#D2n-+Q z;`xgU{{G{a2!d(qGaMRLZOk9;W}xm@wg*SZ_!*t-cgbo8BQ6M57wvo2c8 z=X5ULIGR+&)zwO(Phpx5z)u9*6qGZo!|Z51#<1~Z-r6Z^^U-tSrN3)-C$7Pj=n6^= zK{=cW5Dtf}Psmv0g8I%@&(H$6K317VgE2R&> zz$!Ocu!DI@t}PEu3DPEMTNr(+|q-m3kkh?IEE?ka{-5=I=%Nnm;k+pj3dvg9Tq z;kMMfIom!kG^vOOH7iJ`c0N*Ycw9jiTsO%ki?s)Sz7n0Tv7ylQJjbh|Zr(OllyQ(@+`P28_uuNtR-0Kg<#C_^04n~5E4DUdJ(%=r`y zICB%1!~0qXxafheP7zu>Z+sqhaa41x&_9yO3C}zDxh@_qxAr|op!{}Mci$*RgMRo{ z>}R#abF-%&_||esygnVvFKKA9=Z*n}>9jYkux)5vzfA)^Xaj}lW^?*>al^q12pNXa zB^(x$j3&cL+kgIwXI_A9VZRdvCr3(4X{@66DAOHma&>KUj%oey(> z9|hwE_UX8UuI_P06PPJU?8W!hdLOIn&pj&ZJ9L#ZWq0XyGf>No)zHBB8cVD|cC<%d zu=k*31P*I4x>ZwN@{|}d85)}=0Gkv%9pt#%xA8XI8cExmsR+D1Xb8_qar+JuQ{|gD zcyA&pX%8j{N>V?+3L8D#hzjS|1wWsmO@e~){ex4)oz)$}9n(UC22az5Gmw*YqZmld=jTkz2gq&)$cto@scpP(#m zY4}aG8dy#H`nSS8DxMj)$A!0vvNA4yi2vEynO)D=*jR9AXwexPNG6tIG_98}W^GV@ zJ0C8rZEl`6EZ&DnQg~L5iqAfN{P^k9k!V!Pg_w97!}f1nnd`U_xMcn9!M>jL&rBR# zUIJt#cu{*KVkofu3eur3{BZmRfPnx{PFy%%zH{JhnEUn(AgCMJdu^;s@()wVoT2?q6Nb8>KG2&0L#G6!Wg-iLy z=vtbNChkrrM#yW4y|j@fe;SbhAl7=T$GFLmC<3=jM1>;v zjO{_q^DB08dhLFr7!pZ@L~D*=eqUc79I1Hci_B)!`y8a~qF!FYkuUTkS`*D&w?LG*)6V3N#Etg{9r7S0d$R3ha4leOf=Gf#uxgo$YUKn^lt3!VFz39uH9Y64FPVCQ9?a<+YLHBV zZKeUMwqyxWAxBtZnL4*pZbN?AyVE0-qbgHui33IOmh}wb#0Pb>-SC2LY(JW`)8nI@ zfd+dJ7AaJfc^<*=4;o9Ulq9I5H8&;NBrFv%MRUzBWiYtz4W6+|;Jq zEAgKz{}r@6pBK-fRFu4FQ4!m*9BPi7yDgFB#i;fAqzQw3PzB+i65gG4thiUhtZAuA zF=&Wnn>TdX*jXT7WC2|xW?XBrwL-FfM!oj-sx|x>qPaArlDTmT5fS(oYGd9LU^o`5h=Vgv!`VO+yn;aLr{#(+;GQl%Akrw9$!D@v)5 z*2@*&Ci|x5l*BKNlC8dnbFtgu9 zpr1aOhvv^9DJHtPAStARi=-vkCAT3VWNgGqkTtvF-yONY)96fhBPKz1Db{SZ-}* zYlbFiiWYsm8kKVN&GmIX0`_s}`$V#k85yvQ`aZBd;grA$gEY?qy@e!BztXo>U>8Co z+e4)|l(km$U?-l88da%D?*%{uL56hF2kU@XLLt}}z-i6Q3SyZ6Viik>f1j9qJ#cxf z*l&GV2ENMTU4v55*O^SDYyQY}ASm3Nkg*_?Q{Yj_#|y?ds8VP8d>?T}H(e3-(n- zQ1JWF%a)IbaxOnpiG#&6|0vaBo4rTxSjy>`@o3@-rJfy~_NH9)OzpBvm(=1ln-P~G zPEjk~%pT8N4xUbyD@{`mc#cM-bguDn(lEhRBIq zxpJ3Xw&t0b;XaSV+h%}VDa@x|%R<#897AY0f1kAIqgp!M^%vS3I`coP-Z>=0Hj5kH zxYumy6C)0OpXmE=Fsyex$=N)gA_rjOl=z^R-rWx2%cDAFNmG3JDhz0`$nh#^#avdL zc+`}#{$SG(V>Ff9odv=wTr7wOoJ>Bctw0tD@Q7Q%L~1KlNks*!I~UU-l$4Kw3F%x8 zSYdhbkwwMZtj?y6UAyhFsT}YH4=p<^5>OCz+PHPunU$#B)L5j+GKqsK4CI62(N^xP zpeO-Zb0-f-@@)2BP40;VSIE$6^-R9f`pXal?DS8CEH6#uV*nC50_$kZ%(ueQ`8X3g z#=1pD(YE7B`v`85i)TO9M;FRoiw9ok8(~LCd{~y~l+K^MSOQgBsmTptN_{ykRf9z2 z2>bNFL~sx%8GOLMl++aWt5J}+1!~ZDm*&$!+xexo&MQSI*!Garuot9yuLN^XT4*cx zeS(5E!9f_$=*U5htNlWzv^d|{FGL%s*3~PsU5!FXX?MWMM6HNL3h>E7{#M=Q8BhWG z0H7Mc*1#9)bH~H)Atf|=fksJQ%^(yj;&FwOh-LhU^>rCFRA($PtVr~cmHSIOIZGtH zW8ZYkr#}z^$qLO|Z7?hEzXf68&XNt@da;x+&x!_jyku+*mt@}U4)vYYXReq|sTO*k zN9B%Pn1pDdzqe}cxU zy6WX}LvAu`{&VMH4C>v_{sSf&xBul zjkt*J@{00maWB6lhF%a!pxLj8qNCYZk9Sy*+v_*O)>$Je<=@G}DZ29&%Y_zDn&dA- zh#Q-35TGo46@-hP83LybX#6WSf2ijux%eHI&b9_?N*`K2{qo)rjwD(^NhVJw!mzz> z)9QFoJ#e(P(^66hdr=yoEXeZk+VX+4+7lf`<*R}T(vhETorJxc)-(Mw@@sa_{8vTy zO5s(U^$-rwQiZqMM@L7u0C|a}<&GU-FnMQlARhrVh7;5tmWDm{=DU*tVqNKF33Cf) zH)bsFGaM473IxK@H0lbk*+jVoS|k5RboZP8%IHZ^(c8t(4X*oq_KGyja(;y$S< zd`Pex(|%{V+w-$Ne~^r33qpR@_!rN?i%)T!aGWz?!q41Hd3odha?uo|VuPt7sQ>|D z2Y9^j)e@r9!Q2N22M7OC4i1j88OsBxe2UYsX}emVt1sm9&?o|YBDFka7oN}HPqO%}LsfM` zP$plQD=(XXoF5Lt0~!z#LeAKS_wOHr_AN`e;E7adIJ`Cv7rGz>jcuyenZMHdbU)Qb z>{!YObx~2=cNRNL8F(1hOP-E@ZVC!{dSYu#DXci5yGrGSRq%kExahBRJ0L<+h~YrK zN}RUMJBk-XoUVkTKTBJjS70cT!e@Qqe0~ScK)udJ0_+4&^U)fqe^N)WUW|Pwp4xZ2 zrTmt*y@SuX?_%ya2DkUk%ZugPik4tZtPPSARf3fyjGNUm5*agIZT~atYr>vUt@JC2 zb#0mo$m-~?yPR~?$|JoCg_m@G90oJE*|{F_0*JMFc~=-U2hq~C)J4EdS+<~KiQc-H_u~n&i@UF=-$oZ4=M&f!Mx9pZ4 z91)6!p{fA+nUP=iYiFA|9WweR>*1jXkFJR(O74|@Zr88ut))~H>{?cRv_s0>F{kT4 z-l~zZtJG7XE0CXl^>>RnHIV3$H&E-eYdt* zPq3vHcXpsN7iGKsC7q|*BB>9}kPVN^FEPtIv)60Py+SH5#j||1$1fX0cTXIgU4mI2 zT=k6Bsr0MAnNPy%#<&t3!j#@sZItpPX&l+s27mwFZcz3!B}M1J#m$YstZYIzB{nq` z>VDYN6DcpOzV8Yc9khIWGxx4?a+sh=<;bEzob~_DB}oUEUtceZlrJyQ4bsGMQbwS7 z7j<2Aq9kGV&SCX<2Ne3UUR{`9d~%zgh`T<`;ks0a8!Y}4_4R11hIL98dSIB9s_N+d z%36TwrH>~*_9&YmMnLh|lmwuD0N)gN?!g&Iwzh15)@5I{ ziS*>j^XJc(@_uDu(=*$$dS43zE|olhe*sm726MM{w+4M}Io!)&M?86Zij-jW)KOki zv?!~YH~DAsPQnQE>R~IPo}NIWeZboqgYU9A zHh`0T$$|!_^E|H(n#D^%<8e^`m!7k`YT4BD1kC|y0WCXP_qJy?)`zE5!vkh&ZfD01 z+Dt|xzu%z~vHS7aZ~O#Xk@+DKA&yo5@$vETU;kSg!D8|sB|6K-L zZsa4%kWI`C;Bg^LTsRkAM&IoW%TJaKM%aL zPQU_Gkrd6r5mUu{LDP=Z?h*P*B3Vs*6CdxFcUcyzY9twzG3|vMV@%N(p~`GrrvuF| zT8I}M@-@cR8&R~<*0;LH-ff(Hk@>tx71`zJYI(MephRq>;pBjnlfaqz*))st-oqpP zPoX`PqAncwIg6c*Kc8IyZT1Kywd0)#U`p+e?>#>y6YwQNEGsO|r-c^li>>RH=XczZ zQRVAYz6|lJwN!~$I~R2*?DLb+YK?JbS$x4Yeuw8vS!M7n_w}7vE$n_`zVsgL%7!{a zD$AnYBmORgmh;_!*q8ZvE@CA&i|DobsQO|~jnUHW&9NmAP$N%MDZ<%TD~gQpqFAeK zNUQHir!Pk;+oIs;x{!E@f0sRpO!eNwXiz1`bC4OWQrxh2JKTAvnPpnJ{E0N za#Y&-G-j0+JDD^G1>gG5(4!6#SFJG8ViKIxE?SGbDHWZ~-^7^cPo&@oo-NCRM7-1J z8vx6tPP(+8S`JdN8?Tcxaqw3ZuB><2dO|Igr=1}C#qD$jug=jq{WL1GL!=@>T^1^r zHq>Sx;(P?o6hfs1(O}5br7g9_=;H2MfvX8zmtA_^ z>b$`|nji{@i35$-+k2LEW(-5z4`Rch$;F8ZC}{T0^Fx5FvBLS1Tkt378Ynf8H8wRh zwsd%HjINc(Qq>%2beT6o8)EiaAjBSCi}4amav8&oZQp~Wih$Mn*6e@b#)92SI%Tdu zh75gmA*Yy)4En9X*s40Ff@qv5hS5>gt6lf$mv)~Uy)pBvV#}uJU*EnvMiqBAWG6T( zXYYSAYk)F#uuxJu0yqdG2C!u(jNM)7jCq`L-YMrs=ad`_Q43j=(IxBG#_ZLAs9{N8 zvB9NU8}Hq0!L_iWHTAJFWre`OB$U7bC+6pPFMA( z@ZPs<6AgfdAAbdwvMsX_g-=pP&V$fN2sHeC$OUB8os01>^^sgdLAN4}^!0tvup3Lp z)2VjX7A;MQ-Q1uc7f`W}hx1?50i5vR>JW3`{sJXEiRcH`pn(Su+PtqltEUD{r*hS< zox1CG)wKCb{Nj|uzmD~sNmnQnP|WeavT9vr{+8ML~gW3d;Da|m<@;_OdQee zk$Ez9CLDQt-thwYF3De?1_cEkT>-``)2XZG)O)7@1<&6Qey!au7vD(*Z0uvyqECz| zo26}N$pz$0A7DfqXHNTPT1X9i&!t12$l$$~j)x+)o2!O5J%#Z!EI;`kP)b}9Bt?~h z_r0qIwZeSMp6?kbV|1kSKQYot-d`^pJ@5GuQ)xLCaa&CMVfM-eDvQBY=|58$2KKoL zaD$+I(?}!>ej(&V%}{Da{M8MtYs*sw@>@#=K5-|2zQB$Wi)F|of@ZX^?wL_>0ShRg zJv51=eax5) z;Pp>{t*f`4*6;Dwn;3u?b;Pe~v-p;BLttOfBZ-vag$WF|l6fTrTSL*f%RuG=+Mov{ z^YH-qY=@@%?}~|aa^PC8?cr{=ZFsluxEA{#A}~H0Ig}ve#!Kf%)MK!P6yB0m5*@#! zzE-~q6ti11!i=EC2jtEzVw`R;jx1FVKdcr&8aDe5JfV}_nEt*+sdZbtPQve);_aL^ z=eJw&LMi_r$ZH0Oe~w=eh+MmjHfC!t(_iZ}mr*3Ai#|IVqxlJx1g@w&icPBmdglyB zW4Fc9jgK^5w;$z=TxNe%C9RGd0&pG0*Fg`G_-pP`f_OpG`0Ty zP}xupR)Utk$<<-n{D_8-Ea2b&hgRJF*rF`4254Kz0H|q!(=U72y3_K`DStOk_vIN0 zR!8>R0y)!j2h5yEK;_Z1?oUCzlqF6Hn&0jO+$k}!Os_X{oThv|TC(lC1SvCk0=%dJ zcbLeFP9|*)6_p%9;lUmJdW6N~iT{m65~uWnrZM(XiG%?*cAf|-&yd0mG$+yT0b}^Z zR9Q^29M7}Gt-p8;VW?J^2ktgFUM3vgWK*$VE&dz!1PCA!5G>?5qw>f`xEpo@00(kD zdWF&kduY$EE8U#s*sLbmFn@7iM;8b6%-`~-7T|L!|5_XLmS46iBCxZV56Tv9Jp6mQGc5(kQlb)w)} zC%n+~VVGw0J{T-ZXPnv$z+v`!ap8s1Qct4cbhaf;5Y2nl5h$Wh2-$C~{;@~A|7Vi` zjFo8!hm#s#WGYD3c(VCXvP6e5i;Lhi!kyzb^F?wmEkEN|qS(u$t5cw#8INg~RZ@)d zG_mkQy)=GjE7#Pxmp!i3c--QNAZYJsL-~!142-@x12Pu%M4}b|GfgZn9(M@Aer1g* z$RWbXZEQjYf-qumeoRG|v*~Q7wTC*QiQIB_y|5BNk7&N9|Zt2zCemk~Bt_ zY%te!mF%}=hx6&ECXJmtqpAE3^2&7xcgg(}7uoq{f!iO-&Yl073z~mow#ocrnyiQm z1ZDa}8;*N~qmTe5yC?-L2y|(^J>h*cwQ%7UU{u;>8u@uM*LaOQi0C^L-({OJ#$JXM#7+S*OCAFlbpSD<~AQ@rD!m!cyHOt%=mpU$;JiY)Ri989rwi~Gn+!DT{bWZBzq zz^QiuAES1B6}8Z&phZ8w8k;4P%l15?qQ>-Po6o2#*Ch#W2>>UJcAReR4tCjMdN5aq z7uIsp&oJlxaKEV$IW9i=u2&kRqGeYRg5`eieLHRUI`QbONJ?{kO?C!&ep|j$?e$hh zI3b&XnEz@=Al7CFmWbeHdM4&aNvoBcZ{9y8Q{AA)S{saxvXVmF)wFEODcT%h$7S-1Jd~nFooGqpes5*!w`0XE__(2V(_V7(wBQlDbUFv|_3%L5=xYdvDl(=rWC_#q7K=Q~7Q z0Gc>{5huX0f=>s?;+}oIKHWTBkJT-QEWamrU5h~7WYMbqIXEbzQl#MzL?Z}*fS-H1 z%R@wNO(X2Wp1O&Dh@f|4Ja0BjIZXuxg0>ZmDoh?kWOO)7AD>BM1jrH1AAT*?+8TyJ zO)F-A77h^<^v)C<*5HyHD`}*?AswKL%{W?-e>TLNjU`NaH@TUD_86FRYne#A|4vat+i~t~O)Q=t& zuV0kZ@8sKiErY5SXtjcZgXx?FX=nm~v&~V5W`_YRLx<4ioal!S z&%#AuE?MF6To(91B_p`$MpB5=@cRwPo$sscDL#>TkttH6A(+g!J0`))S>d>$gs>Ob z5#3pn%h{ZQyrh^Cp~T`qo6xwhyW+qP-;whv-5!H7R>;y@AmD2Ux{Oc8weiB9{J=mD zp!RllQc_<)pvzKHJVtC=@wS$aR&OmV(QZ3gUUbOPW{{N(4w726pA_N&{dhouoU&YO zzE5hG{hu22fb4d4dmEehZD*lL0F=gnBvq(l;otM*$&*^iq=eU4TsR~FZD-*%pY-fm zGexCo(03sRRX`qk^){Iq=@76`Z!h)*atNybgJfK?0e}*aZ#x1o3eMQ5B!1{YQ-pYv zdeyg36t{V{Vyk!fa3AZ6vP7%aw6m7`v8rz2MENj0CUK7 zBi=NQ-S{IoN!=wc$0qC%N1PzZsT*HZUdemck$nwh-u`Hy97LOs12W5Cer;5*8ROVT zkS;vcs0QCH#0<<(fpS>S@dDx)oE|78EuEp;;suz-)AL422?^h4JZQYBU%xIj z>U%-%vh)b>UPJ-I6^w-fuy7&pCm$a1Pqh^Bi$VtYg~YcVl!4;q(LQB7C*>bsr0*bg z(M@h20%W7Zvd2JE*AK8Ch9(=p_P`6nAmRQ&;dRw!ShASws^^&qD8?q&#podGOlhfR z!T+~jRH``;AitA(|Ngyyl{M%u0hKK|os9W1X=3nLd5FtSNa`c0=32B5e@t8YD}j>| zRmw?81X2+&A9_;g;NWXEg)w5CZ~&nMOkDPY@xhqK2UHes;gzk9P{nm6W+3SuqB zu)3^tG;@F{dwd+5A#z)rDZZ;cFKMBAMS$1+jceuGy!YEt%CoQZgd6ni1Z#q5g`H(3 z@4XyeXdAQ(YA1tW15Nr|J5yamU*# zxTW7{grZX~kUGhC@Vg zQ!50Z-VqDuM$G|{`V0g`BNG#Lx3e!m^41Q>=iOz7JqWe?)w3=@94DmlTRFXQktUn= z$vEs}zei+N0zj*PF#s0;$^RR!3NQlDmIwz96E3q&_zv(Yxa?L)B}sXCWag0e{5;0c zqi&rLA8;G0xkiy0ZgQ;}KT z2-6H{nKzeVlm8MHD@RlwAZir0X*!R&wU6xc1A1|w8Z-p~j+{Y`-qwMGSNbICZ}XEY^_><8@KlQYvC45kpoapPxq6G25O-oIT#5{*2g9QUP*H9Z zXE9zzjX%#diry->Lz8yV@54kmsQpdli9@+t#|!SUSfEr3mzfGU9~dn(x&pO&1R(nZ z(RoJ{of6P!o;rjosxT!41F;>W2}fO*lW;QoHw+YcHsH8e@ZJDduLWwaJfVzeTLPbE*IQ5*U zj^=OR3cNbl%`P9{9LwcY%*nQ8lSj*a=sHJy_});WIcV}vFDpt=^f!D_-MSP+s!_s; z?RjP(GGALWGB7X@)DBn9vE;C&{y&PMn|v8~!UYCe2Lw{;)t2#r(hw=Bb1IQc%&OzMjt8i2{2;Y^_|xEQb9X<`Vg>{Mx|Fh2lu*FTY~XGx zG+*mpc3?MiF{n#Pgfk2bocYTpnf}d(7_2J@G%Z&^RrC}c-3Xl5xc^X2U>wrh*e%v} zM`L(OuLM>=v)VGv`vHbvIbUBj=b25+Z6C$|Ci$1Sz;njW550gu{aEkqE&f98M=;rU z@O&6{dnlwGsGJis)JcL_-#+vf@M*Bky2r8S2x}9A3ov;AKO;5F2$A)-b|%4hAcr@L z!?h-^zr*=iH+611`|fi}H2|W(mw-0twS>P*9ylct0g}tvX$;`8^BCXqQ>^Kee-V!q z``@I1hpH>pnr)faH4lO0_w7(8*nLzUjCtTFu)e@mPLxB5?Xsd9Z+tRHv?*$e+}7Z+ z(u_}BZII@M=F71#c>V%-etBhTtIpa^m07g%x>;5Rx6n#^)xi?Yjry*e- z5@~|NG3urNdWZ&7VlWT+ct8A^S@3&kaN>Xha@vU#pBQG{J|kF9u8)WUG4A#(oFU z-)tIuo}Q7|Y8F_^8zhx`QaS3CsB^pwoLA##5Q(IGvms;8qqP^a6KM4JX+buu7y%Ya zdDmoxvDGmPY=00SG;wkUD448e??56|Elo}TZg<@RVu_g89I;q!JdX`IT!{+a=>S^J zKvw;i9_o2I#KRUZvu+LCkxpTCO7a|}Hz!b>)wqXI!)t;x+XI(%SYAOl3@*RNkgfU>9@ z$VzOYcF6*#>R?6Li{0Ct(@`4yc4MO?pW!zm1~;OWOKM$)C=?s#E5h7(^j;WtWeBehEJ~VGwy!HphJbmrdrC{?x+w^z z3Y3$x{uZb7t1bo`exG?$8QQuyy5;2uzH{E6bNE9+K|zfXe{^hYfJ|Oqo|c=tS_Qg% zcqsn>y9Xq1`K(KMATj~30Jwy|++sV< z)#I1${f_EuYHIK@jV`!?8@$S(4ye04uZ@!gau6{~oISOMdS!u_La65yfoXvj7%x$3 zU?T9D5fKq>hT%dI;H4e+H|K`OD_sXPtK;L!gBu)(GC6X|D`5f$a3($|%iyI5sOZGQ z$%Q=5sa?8{Q`=%tCRu|Hbk@xh<--40X=egW_1^D)%|hWQg;12p6d6i}lqi+4=txNB z%tMBdqLI>II+U%G;TU%^&l-tj+)0VpBtwSSnUeed>YV@ctY`h#UH7irTD5HL{rmmC z(`Wv?_In7Q2HC^`FIQ~%Sn<23)?cQEjwau`w*hlm{pOAHal@m7KUbTpGV^Oa3DHeg z{`$h@EU6`*LTSLr%Ob@?3}x`Gnw%b*)xBrpSe8M4)oSvdWzWR7~0+J@v2}KGwbL>k6doD-&4czX+0$eu8;hSh0~NX zCw?0Z{3?_kbHq%PXvZ=ALVIS1lJXnpat<>--~b=p-<7;+XRVE8rV9k20V*5KbJjg; zJdzcjzETk_1dtjQvg#Ef2Y&4~X0i6E_t}(6eYPi=3D#BHbj3eP$8Pm44{*A{-OdEF2RX?{D-j$6qp8 z9v@8_+IQ<5+v~uFh(gIr zcBQlC*@-ByJqzF|jJ>uWW7O?>`GG~Y>$y&Al~cMv(ZM>2Pa5&7(q9~|S1>8n#fs#w zaW4Hj>IZM^&c?)JktvZk-%o_GZ*A!8zEOkQ#xA|Y=aSxQ^M`dDxgKfmDb0La_x$&u zvx^be+;xIQv7_hFv)eZoX6}`EevYu@FWpt$W>7emO#hIuS#Yt?#-kDVX<7-#L!qDkgYtlh`ZDel^#)&l!=e+yd`-05hj zjb%!+MRWiCN_lr|NHPJXTl&^pC0%^QHnt>P*{SoIGTa{K3B_g=srFFfOUkXn2i>PU zD~@>TGG{2|Mwl(`NO9WylY7pGOlwq!O4*At%2CGPheE7J1k~1n`=aeWwM73nENEsV zqv)%)>~|f~YOSJ$wOg%=c+}@3z_i4Z=(vS9Kg$+;aw_=N+lD{)5fi9~H}7B=s_pE8 zekoEDJmrI-h=^A>l5skdA0}G=BFlasquhsG;wjycVnZaHl3`)0%Vqt~+No)A*Z6gU z>md75iK-^Gx&t3lm)ttP;q_0BR%%W2+2r0kE*|KC(L%d5HInSX7SZ;Wf+qbqg}KU# z0sitsDTy-$q0cR|ilxc^NlS4x>?B6cA2yap4npbNLl}3)=tJZ++F=A%yE%d9N!_{b zri({Fd;Wl{2dcP(x3V&QiRS3P^ptFbtmT}}pX=hZuB_%Kwk&F|GESD{8-=69WD?W_ zQ!m;v#tDm){B2B^xrE-U`9 zy7=VH-NWv_kHS>i2?K3WUFzlKliq;N5*si~2wK4`KU6%lA!{PFv%e0knG zT^5Ve4)%rDGWsr(Ct3l#6nX-euikJ#Tl@NKOmjcf=_SM$kL|1gEX}d)@uj*r#S#}| zQ@3R#s*4zIN1S`wss(8`6{4%<%tY90JKcnWm#?y{zO@N!a|OXEtlrYfkd<6l!^3Sc zNr?G2r5jKGj6`E}5KC3Sw)xHX+UYZeUm?GDN1d#0{3VD#X!@lnU?D}`^mW;^7rI$l zS&hCs9ez71N)se6M6|O@UhR~ZUz<~;pSS!;SF`cK{azP7MZpyqv*0ELqs%Cx3EzvO zbkGicoM;dy2s2c)V;l~GvvAP-$`d?#2h-=Eu=##57^O)YMS!}UL`+2#j%!`nCTtq zM9y9_#CsmcSsOJo2PAij7=%vZ4Eo%9dLw1wgg;LK@FbI)80U{<SOeUq1OJ&M7%Drb4$rm;m(iRl@dEE7h*;4Vo%GlNf(ONH_sfOKM zKdeSsGhF@pt8bt+0>#*3?f1n0mGjEjK?yIcXlK@e)A_Z^o#~f$TOq%OA|6UP^E_Gh zjG4zr^OxaQ9=q~#2=Zt+YP`_r$0Y4iM6e`H^n<*CVSm8ydJ%@hdf38f!D5(1K^S@@ zxq4(R7B*E;i0oLm@R*yCklg*sN1C$fALt?Q3_YS(K_qB^{=TuI$IDBy zu3zM(M!okIqGVkBZSh>)gdLYze={byx(HUo(uzH;$Mn}7_F%kQdO7%E;#2ESNplL| zaiF$00i6m50-#$Xnrh4T{b@RHlCf5&&8N6ibHQn1XgynDmbf6CXHs10ThPskri&F{ zWL>cr(N#;|Lgcxp-P*%P?SjJ7F--en2z#s34&1*4&kS_(mSG#q1#7BB{1TtE>&{h^ zxe*K6c_8}fjderX^;9 z4eN6$#4qy}Yj4Tf@%O$)8q_cxoq_Nz{}yp4pluqzpWcO>z%}M?DW{KKrPh+6apU*; z_7!z#DvZhVq2NNs&^ZsA#FPH@B-Ld;l!d{SYIY^ZCt9y_+nmmkvvNj6W2SfmCd~#J z$#|lJB^B=7Vp0<{h8l13$7Q${Oqy;!f{F9&(d>d9@Vo>!R7cB7rq<&{zpP)l;wo^D z<9@rMe$rDU-G$(tBk4sZkPHBue6vi8Iw=X3$zZpAO#j{g!L`wq5eWwO}{wQCU5 zR>kYx{%N{q2V_3OD()I;KY$3Hhd*tY`lTplG)QBdA8zqKQY5reihPT=>z z_kc@4(0-sy(!|S_mfFTKFP@WB&BDVbPLKH(`u<8UeFiwd7RW#Z)1nVa6@>ubXY3bP z8zo5sIlq_r$*avxaH0FnvMKQ(LaNU}6>$#!eixyPF)?@{@S{Ac_`SE=H*NNt`10h( z^fUfFKI2gAJ)#2>Z zyGppD?NzUBc0lQ!VhqsaRXet`!RI0!yX#C{)sy;=5F(7BVI%WzpXKH5C3HF(47Otl8#t1L* z23mac(Aj0)O7}OY_(eQ=wEeKiT`kJ)g$ClKfMfPF@>lVBNUlm+O>25Is3*CqWZ?%p zsXR~(aRtH*a?C1EPj0)>9#dxnY~l$jl~X`qcznK4TRy^sSDQb#-;Iks~kX#9Q>= zPR&@*JmS{qr;So@dT)Wg*NJA`0D~&zoz{bY3HS_vQy#Qq#@Wb(Q(C?aq*I#+0#HZ? z2}}AQwc`6hNo$qap`4h*BEuE)f%pZNuJVOfR@=P6&WURs6!q$<<|N=0ph)-Y>2Z{u zo3w$fG^3Kd{@QYBusFpGlYR1lD}92D^#L${L`AQHKRX88BF#ZCZst~8TrILay2zXY zv1ti@003@T9p(keowk=x5n~T&QY~DX$g;`dk7iUlLVllHD}KrY2K3ZQHrsdHCcITI z&O}>V+h?ll3gNMUV3=#)d>OGXSAZ|5t&6idcK}%`ym@m7iQgb<2bo1@DAfR)R01ys zHfj~!3sH}kOTdmLu|$})KTvT90;+dzw$X4v`* zNN|yA47K?mJEY5}gd|OJNBtx*ui}#`zZZLoF}0rbzHdC}GhEc?!=M=xQqen*41o08 zfRw*7`i34GV*zHr7UX#L8+3qWn(x^5lzM*d3|gq4p_yoa;dN#v;QNp9`W@cGrl0F6 z^#B?IRn>Lpkko^}{;E#*LI=7Byu7_NX+0V+G1l!QqcYo*=i%XT1JE75L(x+5BuYsypB*0eTLt}F*jV6V zPOWgZCLkR;YDdPscp{6T`oge*cyv;l)2ZjCd!zZl&Xe*UianDh17C22ri2@^u&}U& zzOY1MYhCY6_VwER#15B0}x<))#JyH!+=Glqc9-4bwKnCqM6Gr zz}!piFssPVi!doUo4IKH3U+}BU&*$%w&fQ7m~HGpNLfvP|I%!)K_e3FeM#yH{L;Ik z<=mS?tZi+VG$2DudE4c>(+6I1b$z`pDmPS9Phl&W22S^4c3*C#nmb@OlU<7L8Ceuj z7#JGjWWoi01NBijTs(O7=FMazP)fo5Q%y~hK)EFNvd>Udqf(yjbmTI-pbiQPfNxy< z2_R=HFfB~|`jgBumsC0z&|P=JS(oru78R|weHg$*KzxWTx6f-qT95)5RF2GMWL*~7 zy_*lAZY!%q-N8p~j-^PzK_oViX)n)5R+G>Q9$XRl6FFeB04KZ4e6|YffK8qqsi1HQ zYc`}COH*1%{1#_GQ(vDGbTogU@Fu=AJ{$BZMMlrDy|7Vyj)g|dDwpzU5fnZ2!j^!w za2MH6O6G}z-?+Iy(@eg9{ZIvbr8dI6q^9Bb0%!h;`MD8vgdk7Vn4TBGK!g4r#B|Yn zHo#~N?76K~{p!^N_@jtPZ&r#M+mlR68)yY%$*_L!P(z@FT)ZCg#cJfSE} zO;3-Z|E(jPdP2J^`C6>3H47eq(QodHVOc|9SlQZ+;|R#Qb!$XPn-9~1Jiz@1B26_) z>ZC~*)xt%5_u>YKS1?qxA?^q^w$7>GBgPLy1Kx7?AGYy}60qb*nMI^!YxMUMT?g}m)uZiRY*vSZG9#1=G9o3QnSz& z9aQB2hO7dC3K2h(aT5qoA~y-!1Ek0&D6t{30Xyz{&7jM>M;D^a{I1>LQ}hd1njevz z6q}H0nDPx%Gm85)L$y(8r}9it*J6p7KDt^$37!U^aNj)KPr>!alpF3JM4<=<1_t8H z-vd3jyRYxcv#wDG73=+)ngEckL%k54(Rwp9G@V1X#Cv2o$6huF$@*`wvfVM1*&c?V z-{<@YUY%q?`_m=t%9T)F{sy|Rf{&N`@ACEaC5U4{c46P{Mb}?|0s9_D zM+v6>{{G&ZDJdx>BWJR-gX@q8fJ{xT*K}e+eF#Du-NqF&YtgG#SzjN%jFsExD`Ot( z;f7~NA6wtgvU8)K)CK%U)s;NIe2U~nK_CaJE0nypQT)HH>H_fl61|YbRjq(Ja3->4^=b3XxK|4z!xqEs^xEon<6D56{D}sIvrS3OUf0ukLIyOI2?+ z5V#c`Z9GgJnVX%=!G18wcRUD*QfgF%8Ui0w?o1Em)HAcPdemQlgUYdHO;4T@6JYgk z2;v!Utgfr;MGcrWGTdgf>?KD*puZIV@YXF3LMCUJC8Bak1|YTf1upfC54C(@)MOu* zX!oJtfSCGpVTFJN)ICADn4X_wAgmaLJ$fI1KMv*v4MvZk{v-x*q`<)P={z}HCO&Y+ z9*d&|ZAvO({$`%3rrKgOu)@Hzh6lyYt}?*^U4I%r7GtJZeonOnk;mzA%KTx_(f}Q_ zf_5ZgH$AJkCfy6ADtBUI2h4Pnh9h+YW&((mb zEQTS=%7fghCQir=@Mc;>rKG+eQ~|86yb-?LGlx80Jn&1bdRRtAhTr**v4KQU!sg=9 zJbXBelY_$oM95;ZKxcVcHi6cToX298nH=g*`V#rEeE#P->RW^3$L&l57&J`?B(`*# z19m$PqJ)>S0nUbjz~f@h;w;N1C|QBx4Iu;e77v5~{&!yp0X)EgoE*nCSyZE;PPANV zzs3L|guoE~_b-%a#goGg+>bK?z9DhOQGdp(6y}f6(=i%}5$UCfo z+5s|q2NHvCspjUoop3BS!-oMOEXQW-?FkndptJIwKD;FIt4^`=^|#*IkA2D$4g1LL z2(cKCLXd;7VVR6YNg+&8*JZwUkF8k=#h8GD@F`EVK(uQAfX!uziepmv*J^g|Mngk` zyO;9Jmxl|NGOrL&Q>$>_9TW~ECGPmx`J)nfYT+!VloHibi((eoN$Bv&M=!mY=YM#A_d9_v`0axjCsM@#RB{ zS2-stD*CBAR9Zw%t|j0c0M749ys5LL2s0O{Ik!FDLNqE=Boi4VERBeB9r*Xrxk+az zdxR2RT4uY#elt382=;ZmVAYEki-^`Ck!O}+ zM|JoE%;%xul3m8w9eGOBA-TJ+tQMQj?O`Hw4f~yrddWW2RN^2C5PL=-Rab+=?5_G8 zL;isop#^#QNQhqdf{>7WJ>?*cJ)I7n7o9cI8nfQmfW#>ai-6TT&^dkBf6;blhtZynR%1$pGn0N+@9$fKMy{ z(=!8*Z*WQ=N!cr5X&*%QDC%*$8npyrpj?M6V+lf&>@BFdad=ilNWkW)0}<_M=g;qo zOweML&#!S20|D5~B$v+e4az>ZU^02kFO?mJjxuf>cq@Z>Oif^$y!~&j09B)5J+yg~N{y z*ymF?95`Qa3hanUI66702=Q%OI{K6qCrmGa%(6`UNr$ffhuLI zd{=S4Lf_QXu^m>KCL%;cNWzs}kLmVO}VhIA3`QJ3b6nL5`_P+UrqBvZQat(U4nb}TH@<>m=5mO(a73saq+(ZD+LEx literal 0 HcmV?d00001 diff --git a/learners/setup.md b/learners/setup.md index 1fec3bdd6..534494f8c 100644 --- a/learners/setup.md +++ b/learners/setup.md @@ -72,16 +72,47 @@ Remember that you need to activate your environment every time you restart your 3. Install the required packages: +In the course, you will have two tracks to opt from: **one using PyTorch and one using Keras**. We recommend +PyTorch for the intermediate-level Python users and above; and Keras for beginners. + +::::::: group-tab + +###### PyTorch + ::: spoiler ### On Linux/macOs ```shell -python3 -m pip install jupyter seaborn scikit-learn pandas tensorflow pydot +python3 -m pip install jupyter seaborn scikit-learn pandas tqdm torchinfo torchmetrics torch torchvision ``` -Note for MacOS users: there is a package `tensorflow-metal` which accelerates the training of machine learning models with TensorFlow on a recent Mac with a Silicon chip (M1/M2/M3). -However, the installation is currently broken in the most recent version (as of January 2025), see the [developer forum](https://developer.apple.com/forums/thread/772147). +::: + +::: spoiler + +### On Windows + +```shell +py -m pip install jupyter seaborn scikit-learn pandas tqdm torchinfo torchmetrics torch torchvision +``` + +::: + +If you have a GPU, you might benefit from following [the official commands from PyTorch](https://pytorch.org/get-started/locally/) +for installing the `torch` and `torchvision` packages. + + + +###### Keras + +::: spoiler + +### On Linux/macOs + +```shell +python3 -m pip install jupyter seaborn scikit-learn pandas keras tensorflow pydot +``` ::: @@ -90,17 +121,26 @@ However, the installation is currently broken in the most recent version (as of ### On Windows ```shell -py -m pip install jupyter seaborn scikit-learn pandas tensorflow pydot +py -m pip install jupyter seaborn scikit-learn pandas keras tensorflow pydot ``` ::: -Note: Tensorflow makes Keras available as a module too. +In this course, for the Keras track, we will be using the TensorFlow backend. +[Keras can also use either PyTorch or JAX as a backend](https://keras.io/getting_started/#configuring-your-backend). + +Note for MacOS users: there is a package `tensorflow-metal` which accelerates the training of machine learning models with TensorFlow on a recent Mac with a Silicon chip (M1/M2/M3). +However, the installation is currently broken in the most recent version (as of January 2025), see the [developer forum](https://developer.apple.com/forums/thread/772147). -An [optional challenge in episode 2](episodes/2-keras.md) requires installation of Graphviz + +An [optional challenge in episode 2](./2-keras.md) requires installation of Graphviz and instructions for doing that can be found [by following this link](https://graphviz.org/download/). + + +::::::: + ## Starting Jupyter Lab We will teach using Python in [Jupyter Lab][jupyter], a programming environment that runs in a web browser. @@ -119,6 +159,37 @@ jupyter lab ## Check your setup To check whether all packages installed correctly, start a jupyter notebook in jupyter lab as explained above. Run the following lines of code: + +:::: group-tab + +### PyTorch + +```python +import sklearn +print('sklearn version: ', sklearn.__version__) + +import seaborn +print('seaborn version: ', seaborn.__version__) + +import pandas +print('pandas version: ', pandas.__version__) + +import torchinfo +print('torchinfo version: ', torchinfo.__version__) + +import torch +print('PyTorch version: ', torch.__version__) +``` + +This should output the versions of all required packages without giving errors. +Most versions will work fine with this lesson, but: +- For PyTorch, the minimum version is 2.1.0 +- For sklearn, the minimum version is 1.2.2 + + + +### Keras + ```python import sklearn print('sklearn version: ', sklearn.__version__) @@ -129,8 +200,12 @@ print('seaborn version: ', seaborn.__version__) import pandas print('pandas version: ', pandas.__version__) +import keras +print('Keras version: ', keras.__version__) + import tensorflow print('Tensorflow version: ', tensorflow.__version__) + ``` This should output the versions of all required packages without giving errors. @@ -138,10 +213,21 @@ Most versions will work fine with this lesson, but: - For Keras and Tensorflow, the minimum version is 2.12.0 - For sklearn, the minimum version is 1.2.2 + + +:::: + ## Fallback option: cloud environment -If a local installation does not work for you, it is also possible to run this lesson in [Binder Hub](https://mybinder.org/v2/gh/carpentries-lab/deep-learning-intro/scaffolds). This should give you an environment with all the required software and data to run this lesson, nothing which is saved will be stored, please copy any files you want to keep. Note that if you are the first person to launch this in the last few days it can take several minutes to startup. The second person who loads it should find it loads in under a minute. Instructors who intend to use this option should start it themselves shortly before the workshop begins. -Alternatively you can use [Google colab](https://colab.research.google.com/). If you open a jupyter notebook here, the required packages are already pre-installed. Note that google colab uses jupyter notebook instead of Jupyter Lab. +[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/mimer-ai/deep-learning-intro/scaffolds) + +If a local installation does not work for you, it is also possible to run this lesson in [Binder Hub](https://mybinder.org/v2/gh/mimer-ai/deep-learning-intro/scaffolds). This should give you an environment with all the required software and data to run this lesson, nothing which is saved will be stored, please copy any files you want to keep. Note that if you are the first person to launch this in the last few days it can take several minutes to startup. The second person who loads it should find it loads in under a minute. Instructors who intend to use this option should start it themselves shortly before the workshop begins. + +::: note +Training deep-learning models can take a long time if you are using Binder and you may need to reduce the number of epochs. +::: + +**Alternatively** you can use [Google colab](https://colab.research.google.com/). If you open a jupyter notebook here, most of the required packages are already pre-installed. Note that google colab uses jupyter notebook instead of Jupyter Lab. ## Downloading the required datasets diff --git a/profiles/learner-profiles.md b/profiles/learner-profiles.md index e5d4ae372..eef535ec0 100644 --- a/profiles/learner-profiles.md +++ b/profiles/learner-profiles.md @@ -2,18 +2,18 @@ title: Learner Profiles --- -#### Ann from Meteorology +## Ann from Meteorology Ann has collected 2-3 GB of structured image data from several autonomous microscope on balloon expeditions into the atmosphere within her PhD programme. Each image has a timestamp which can be related to the height of the balloon at a given point and associated weather conditions. The images are unstructured and she would like to detect from the images if the balloon traversed a cloud or not. She has tried to do that with standard image processing methods, but the image artifacts to descriminate are somewhat diverse. Ann has used machine learning on tabular data before and would like to use Deep Learning for the images at hand. She saw collaborators in another lab do that and would like to pick up this skill. -#### Barbara from Material Science +## Barbara from Material Science Barbara just started her PostDoc in Material Science. Her new group has a large amount of scanning electron miscroscope images stored which exhibit several metals when exposed to a plasma. The team also made the effort to highlight solid deposits in these images and thus obtained 20,000 images with such annotations. Barbara performed some image analysis before and hence has the feeling that Deep Learning may help her in this task. She saw her labmates use ML algorithms for this and is motivated to finally understand these approaches. -#### Dan from Life Sciences +## Dan from Life Sciences Dan produced a large population of bacteria that were subject to genetic alterations resulting in 10 different phenotypes. The latter can be identified by different colors, shapes and movement speed under a fluorescence microscope. Dan does not have much of experience with image processing techniques to segment these different objects, but used GUI based tools like [fiji](https://fiji.sc) and others. He has recorded 50-60 movies of 30 minutes each. 10 of these movies have been produced with one type of phenotype only. Dan doesn't consider himself a strong coder, but needs to identify bacteria of the phenotypes in the dataset. He is interested to learn if Deep Learning can help. -#### Eric from Pediatrics Science +## Eric from Pediatrics Science Eric ran a large array of clinical trials in his hospital to improve children pharmaceutics for treating a common (non-lethal) virus. He obtained a table that lists: the progression of the treatment for each patient; the dose of the drug given; whether the patient was in the placebo group or not; and moe. As the table has more than 100 000 rows, Eric is certain that he can use ML to cluster the rows in one column where the data taking was inconsistent. Eric has coded before when necessary, but never saw it as something he needed to learn. His cheatsheet is his core wisdom with code. His supervisor invited him to take a course on Machine Learning as "this is the tech of these days!" his boss said.