2. First steps with Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #301E40; border: 0px; -moz-border-radius: 10px; -webkit-border-radius: 10px;\">\n",
    "<br/><br/>\n",
    "<h1 style=\"font-size: 45px; color: white; align: center;\"><center>\n",
    "<img src=\"https://raw.githubusercontent.com/HumbleData/beginners-data-workshop/master/media/humble-data-logo-white-transparent.png\" width=\"250px\" /><br/><br/>\n",
    "Data Analysis with Pandas\n",
    "</center></h1>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ***Note***: This notebook contains solution cells with ***a*** solution. Remember there is not only one solution to a problem!  \n",
    "> \n",
    "> You will recognise these cells as they start with **# %**.  \n",
    "> \n",
    "> If you would like to see the solution, you will have to remove the **#** (which can be done by using **Ctrl** and **?**) and run the cell. If you want to run the solution code, you will have to run the cell again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\" style=\"padding: 0px; padding-left: 20px; padding-top: 5px;\"><h2 style=\"color: #301E40\">\n",
    "Data analysis packages\n",
    "</h2><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data Scientists use a wide variety of libraries in Python that make working with data significantly easier. Those libraries primarily consist of:\n",
    "\n",
    "| Package | Description |\n",
    "| -- | -- |\n",
    "| `NumPy` | Numerical calculations - does all the heavy lifting by passing out to C subroutines. This means you get _both_ the productivity of Python, _and_ the computational power of C. Best of both worlds! |\n",
    "| `SciPy` | Scientific computing, statistic tests, and much more! |\n",
    "| `pandas` | Your data manipulation swiss army knife. You'll likely see pandas used in any PyData demo! pandas is built on top of NumPy, so it's **fast**. |\n",
    "| `matplotlib` | An old but powerful data visualisation package, inspired by Matlab. |\n",
    "| `Seaborn` | A newer and easy-to-use but limited data visualisation package, built on top of matplotlib. |\n",
    "| `scikit-learn` | Your one-stop machine learning shop! Classification, regression, clustering, dimensional reduction and more. |\n",
    "| `nltk` and `spacy` | nltk = natural language processing toolkit; spacy is a newer package for natural language processing but very easy to use. |\n",
    "| `statsmodels` | Statistical tests, time series forecasting and more. The \"model formula\" interface will be familiar to R users. |\n",
    "| `requests` and `Beautiful Soup` | `requests` + `Beautiful Soup` = great combination for building web scrapers. |\n",
    "| `Jupyter` | Jupyter itself is a package too. See the latest version at https://pypi.org/project/jupyter/, and upgrade with e.g. `conda install jupyter==1.0.0` |\n",
    "\n",
    "Though there are countless others available.\n",
    "\n",
    "For today, we'll primarily focus ourselves around the library that is 99% of our work: `pandas`. Pandas is built on top of the speed and power of NumPy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\" style=\"padding: 0px; padding-left: 20px; padding-top: 5px;\"><h2 style=\"color: #301E40\">\n",
    "Imports\n",
    "</h2><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Import numpy using the convention seen at the end of the first notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_01.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\" style=\"padding: 0px; padding-left: 20px; padding-top: 5px;\"><h2 style=\"color: #301E40\">\n",
    "Loading the data\n",
    "</h2><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see a method's documentation, you can use the help function. In Jupyter, you can also just put a question mark before the method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "?pd.read_csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To load the dataframe we are using in this notebook, we will provide the path to the file: ../data/Penguins/penguins.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Load the dataframe, read it into a pandas DataFrame and assign it to df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_02.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**To have a look at the first 5 rows of df, we can use the *head* method.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Have a look at the last 3 rows of df using the tail method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_03.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\" style=\"padding: 0px; padding-left: 20px; padding-top: 5px;\"><h2 style=\"color: #301E40\">\n",
    "General information about the dataset\n",
    "</h2><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**To get the size of the datasets, we can use the *shape* attribute.**  \n",
    "The first number is the number of row, the second one the number of columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Show the shape of df (do not put brackets at the end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_04.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Get the names of the columns and info about them (number of non null and type) using the info method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_05.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Get the columns of the dataframe using the columns attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_06.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\" style=\"padding: 0px; padding-left: 20px; padding-top: 5px;\"><h2 style=\"color: #301E40\">\n",
    "Display settings\n",
    "</h2><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the display option of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "pd.options.display.max_rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Force pandas to display 25 rows by changing the value of the above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_07.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\" style=\"padding: 0px; padding-left: 20px; padding-top: 5px;\"><h2 style=\"color: #301E40\">\n",
    "Subsetting data\n",
    "</h2><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can subset a dataframe by label, by index or a combination of both.  \n",
    "There are different ways to do it, using .loc, .iloc and also [].  \n",
    "See [documentation ](https://pandas.pydata.org/pandas-docs/stable/indexing.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Display the 'bill_length_mm' column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_08.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* We could also use `df.bill_length_mm`, but it's not the greatest idea because it could be mixed with methods and does not work for columns with spaces."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Have a look at the 12th observation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# using .iloc (uses positions, \"i\" stands for integer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_09.py"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# using .loc (uses indexes and labels)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_10.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Display the **bill_length_mm** of the last three observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# using .iloc\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_11.py"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# using .loc\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": "# !cat solutions/02_12.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally look at the **flipper_length_mm** and **body_mass_g** of the 146th, the 8th and the 1rst observations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# using .iloc\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_13.py"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# using .loc\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_14.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**!!WARNING!!**  Unlike Python and ``.iloc``, the end value in a range specified by ``.loc`` **includes** the last index specified. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.iloc[5:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "df.loc[5:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\" style=\"padding: 0px; padding-left: 20px; padding-top: 5px;\"><h2 style=\"color: #301E40\">\n",
    "Filtering data on conditions\n",
    "</h2><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**We can also use condition(s) to filter.**  \n",
    "We want to display the rows of df where **body_mass_g** is greater than 4000. We will start by creating a mask with this condition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "mask_PW = df['body_mass_g'] > 4000\n",
    "mask_PW"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this return booleans. If we pass this mask to our dataframe, it will display only the rows where the mask is True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "df[mask_PW]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Display the rows of df where **body_mass_g** is greater than 4000 and **flipper_length_mm** is less than 185."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_15.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\" style=\"padding: 0px; padding-left: 20px; padding-top: 5px;\"><h2 style=\"color: #301E40\">\n",
    "Values\n",
    "</h2><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get the number of unique values from a certain column by using the `nunique` method.\n",
    "\n",
    "For example, we can get the number of unique values from the species column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "df['species'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get the list of unique values from a certain column by using the `unique` method.\n",
    ">Return the list of unique values from the species column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_16.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\" style=\"padding: 0px; padding-left: 20px; padding-top: 5px;\"><h2 style=\"color: #301E40\">\n",
    "Null Values and NaN\n",
    "</h2><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you work with data, you will quickly learn that data is never \"clean\". These values are usually referred to as null value. In computation it is best practice to define a \"special number\" that is \"**N**ot **a** **N**umber\" also called NaN.\n",
    "\n",
    "We can use the `isnull` method to know if a value is null or not. It returns boolean values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "df['flipper_length_mm'].isnull()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**We can apply different methods one after the other.**.  \n",
    "For example, we could apply to method `sum` after the method `isnull` to know the number of null observations in the **flipper_length_mm** column.\n",
    ">Get the total number of null values for **flipper_length_mm**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "# !cat solutions/02_17.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the count of the different values of a column, we can use the `value_counts` method.\n",
    "\n",
    "For example, for the species column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "df['species'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to know the count of NaN values, we have to pass the value `False` to the parameter **dropna** (set to `True` by default).\n",
    "> Return the proportion for each sex, including the NaN values.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_18.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the proportion instead of the count of these values, we have to pass the value `True` to the parameter **normalize**.\n",
    ">Return the proportion for each species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_19.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Using the index attribute, get the indexes of the observation without **flipper_length_mm**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_20.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the **[dropna](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html)** method to remove the row which only has NaN values.\n",
    ">Get the help for the dropna method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": "# !cat solutions/02_21.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Use the dropna method to remove the row of `df` where all of the values are NaN, and assign it to `df_2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_22.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use a f-string to format a string. We have to write a `f` before the quotation mark, and write what you want to format between curly brackets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "print(f'shape of df: {df.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Print the number of rows of `df_2` using a f_string. Did we lose any rows between `df` and `df_2`? If not, why not?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_23.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Use the dropna method to remove the rows of `df_2` which contains any NaN values, and assign it to `df_3`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_24.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Print the number of rows of `df_3` using a f_string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_25.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\" style=\"padding: 0px; padding-left: 20px; padding-top: 5px;\"><h2 style=\"color: #301E40\">\n",
    "Duplicates\n",
    "</h2><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Remove the duplicates rows from `df_3`, and assign the new dataframe to `df_4`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": "# !cat solutions/02_26.py"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# checking the shape of df_4\n",
    "df_4.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see that 4 rows have been dropped. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\" style=\"padding: 0px; padding-left: 20px; padding-top: 5px;\"><h2 style=\"color: #301E40\">\n",
    "Some stats\n",
    "</h2><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Use the describe method to see how the data is distributed (numerical features only!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_27.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also change the **species** column to save memory space. Note: You may receive a **SettingWithCopyWarning** - you can safely ignore this error for this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "df_4['species'] = df_4['species'].astype('category')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Using the dtypes attribute, check the types of the columns of `df_4`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_28.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the functions count(), mean(), sum(), median(), std(), min() and max() separately if we are only interested in one of those."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Get the minimum for each numerical column of `df_4`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_29.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Calculate the maximum of the **flipper_length_mm**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_30.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get information for each species using the `groupby` method.\n",
    "\n",
    "\n",
    "> Get the median for each **species**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_31.py"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\" style=\"padding: 0px; padding-left: 20px; padding-top: 5px;\"><h2 style=\"color: #301E40\">\n",
    "Saving the dataframe as a csv file\n",
    "</h2><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": ">Save df_4 using this path: `'data/Penguins/my_penguins.csv'`"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": "# !cat solutions/02_32.py\n"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (system-wide)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}