1. Importing Data with Pandas.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/SKawsar/data_preprocessing_for_ML_with_Python/blob/main/Lecture_01.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y66h391odn_x"
      },
      "source": [
        "# Lecture 01: Importing Data with Pandas\n",
        "\n",
        "Instructor: **Md Shahidullah Kawsar**\n",
        "<br>Data Scientist, IDARE, Houston, TX, USA\n",
        "\n",
        "#### Objectives:\n",
        "- Challenges of reading a CSV or Excel file\n",
        "- Choose columns by name before reading a csv file\n",
        "- Choose columns by number before reading a csv file\n",
        "- Reading only the first n number of rows\n",
        "\n",
        "#### References:\n",
        "[1] Data Source: https://stats.espncricinfo.com/ci/content/records/83548.html\n",
        "https://stats.espncricinfo.com/ci/content/records/283193.html\n",
        "<br>[2] Reading CSV file in pandas:\n",
        "https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html\n",
        "<br>[3] Reading Excel file in pandas: https://pandas.pydata.org/docs/reference/api/pandas.read_excel.html"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EmvuNOxVdn_y"
      },
      "source": [
        "#### Import required libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bdrNcRF0dn_y"
      },
      "outputs": [],
      "source": [
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "824w1FcNdn_z",
        "outputId": "999b29a2-6067-46c5-f129-d53a2f0e962a"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Player</th>\n",
              "      <th>Span</th>\n",
              "      <th>Mat</th>\n",
              "      <th>Inns</th>\n",
              "      <th>NO</th>\n",
              "      <th>Runs</th>\n",
              "      <th>HS</th>\n",
              "      <th>Ave</th>\n",
              "      <th>BF</th>\n",
              "      <th>SR</th>\n",
              "      <th>100</th>\n",
              "      <th>50</th>\n",
              "      <th>0</th>\n",
              "      <th>4s</th>\n",
              "      <th>6s</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>SR Tendulkar (INDIA)</td>\n",
              "      <td>1989-2012</td>\n",
              "      <td>463</td>\n",
              "      <td>452</td>\n",
              "      <td>41</td>\n",
              "      <td>18426</td>\n",
              "      <td>200*</td>\n",
              "      <td>44.83</td>\n",
              "      <td>21368</td>\n",
              "      <td>86.23</td>\n",
              "      <td>49</td>\n",
              "      <td>96</td>\n",
              "      <td>20</td>\n",
              "      <td>2016</td>\n",
              "      <td>195</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>KC Sangakkara (Asia/ICC/SL)</td>\n",
              "      <td>2000-2015</td>\n",
              "      <td>404</td>\n",
              "      <td>380</td>\n",
              "      <td>41</td>\n",
              "      <td>14234</td>\n",
              "      <td>169</td>\n",
              "      <td>41.98</td>\n",
              "      <td>18048</td>\n",
              "      <td>78.86</td>\n",
              "      <td>25</td>\n",
              "      <td>93</td>\n",
              "      <td>15</td>\n",
              "      <td>1385</td>\n",
              "      <td>88</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>RT Ponting (AUS/ICC)</td>\n",
              "      <td>1995-2012</td>\n",
              "      <td>375</td>\n",
              "      <td>365</td>\n",
              "      <td>39</td>\n",
              "      <td>13704</td>\n",
              "      <td>164</td>\n",
              "      <td>42.03</td>\n",
              "      <td>17046</td>\n",
              "      <td>80.39</td>\n",
              "      <td>30</td>\n",
              "      <td>82</td>\n",
              "      <td>20</td>\n",
              "      <td>1231</td>\n",
              "      <td>162</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ST Jayasuriya (Asia/SL)</td>\n",
              "      <td>1989-2011</td>\n",
              "      <td>445</td>\n",
              "      <td>433</td>\n",
              "      <td>18</td>\n",
              "      <td>13430</td>\n",
              "      <td>189</td>\n",
              "      <td>32.36</td>\n",
              "      <td>14725</td>\n",
              "      <td>91.20</td>\n",
              "      <td>28</td>\n",
              "      <td>68</td>\n",
              "      <td>34</td>\n",
              "      <td>1500</td>\n",
              "      <td>270</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>DPMD Jayawardene (Asia/SL)</td>\n",
              "      <td>1998-2015</td>\n",
              "      <td>448</td>\n",
              "      <td>418</td>\n",
              "      <td>39</td>\n",
              "      <td>12650</td>\n",
              "      <td>144</td>\n",
              "      <td>33.37</td>\n",
              "      <td>16020</td>\n",
              "      <td>78.96</td>\n",
              "      <td>19</td>\n",
              "      <td>77</td>\n",
              "      <td>28</td>\n",
              "      <td>1119</td>\n",
              "      <td>76</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                        Player       Span  Mat  Inns  NO   Runs    HS    Ave  \\\n",
              "0         SR Tendulkar (INDIA)  1989-2012  463   452  41  18426  200*  44.83   \n",
              "1  KC Sangakkara (Asia/ICC/SL)  2000-2015  404   380  41  14234   169  41.98   \n",
              "2         RT Ponting (AUS/ICC)  1995-2012  375   365  39  13704   164  42.03   \n",
              "3      ST Jayasuriya (Asia/SL)  1989-2011  445   433  18  13430   189  32.36   \n",
              "4   DPMD Jayawardene (Asia/SL)  1998-2015  448   418  39  12650   144  33.37   \n",
              "\n",
              "      BF     SR  100  50   0    4s   6s  \n",
              "0  21368  86.23   49  96  20  2016  195  \n",
              "1  18048  78.86   25  93  15  1385   88  \n",
              "2  17046  80.39   30  82  20  1231  162  \n",
              "3  14725  91.20   28  68  34  1500  270  \n",
              "4  16020  78.96   19  77  28  1119   76  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "(92, 15)\n"
          ]
        }
      ],
      "source": [
        "# method 1\n",
        "# reading a csv file \n",
        "df = pd.read_csv(\"batsman_most_runs_ODI.csv\")\n",
        "\n",
        "display(df.head())\n",
        "# print(df.tail())\n",
        "print(df.shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3kezCVUodn_0",
        "outputId": "bccb0c9e-e12c-4136-8d8b-c483f1b9c789"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "number of rows =  92\n",
            "number of columns =  15\n"
          ]
        }
      ],
      "source": [
        "# number of rows\n",
        "print(\"number of rows = \", df.shape[0])\n",
        "\n",
        "# number of columns\n",
        "print(\"number of columns = \", df.shape[1])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "l31EuE1Hdn_0",
        "outputId": "b1ea11b1-fb12-40f6-8ce7-4d9e21a406fa"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 92 entries, 0 to 91\n",
            "Data columns (total 15 columns):\n",
            " #   Column  Non-Null Count  Dtype  \n",
            "---  ------  --------------  -----  \n",
            " 0   Player  92 non-null     object \n",
            " 1   Span    92 non-null     object \n",
            " 2   Mat     92 non-null     int64  \n",
            " 3   Inns    92 non-null     int64  \n",
            " 4   NO      92 non-null     int64  \n",
            " 5   Runs    92 non-null     int64  \n",
            " 6   HS      92 non-null     object \n",
            " 7   Ave     92 non-null     float64\n",
            " 8   BF      92 non-null     int64  \n",
            " 9   SR      92 non-null     float64\n",
            " 10  100     92 non-null     int64  \n",
            " 11  50      92 non-null     int64  \n",
            " 12  0       92 non-null     int64  \n",
            " 13  4s      92 non-null     object \n",
            " 14  6s      92 non-null     object \n",
            "dtypes: float64(2), int64(8), object(5)\n",
            "memory usage: 10.9+ KB\n",
            "None\n"
          ]
        }
      ],
      "source": [
        "# checking for missing values and data types of each column\n",
        "print(df.info())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V8hS9RPfdn_0",
        "outputId": "26c9a813-0bde-4bbf-88c7-90590af27291"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Mat</th>\n",
              "      <th>Inns</th>\n",
              "      <th>NO</th>\n",
              "      <th>Runs</th>\n",
              "      <th>Ave</th>\n",
              "      <th>BF</th>\n",
              "      <th>SR</th>\n",
              "      <th>100</th>\n",
              "      <th>50</th>\n",
              "      <th>0</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>92.000000</td>\n",
              "      <td>92.000000</td>\n",
              "      <td>92.000000</td>\n",
              "      <td>92.000000</td>\n",
              "      <td>92.000000</td>\n",
              "      <td>92.000000</td>\n",
              "      <td>92.000000</td>\n",
              "      <td>92.000000</td>\n",
              "      <td>92.000000</td>\n",
              "      <td>92.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>241.021739</td>\n",
              "      <td>226.891304</td>\n",
              "      <td>29.217391</td>\n",
              "      <td>7617.880435</td>\n",
              "      <td>39.145761</td>\n",
              "      <td>9560.043478</td>\n",
              "      <td>80.414239</td>\n",
              "      <td>12.217391</td>\n",
              "      <td>46.043478</td>\n",
              "      <td>12.847826</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>73.548964</td>\n",
              "      <td>69.194813</td>\n",
              "      <td>15.413585</td>\n",
              "      <td>2490.710229</td>\n",
              "      <td>6.264166</td>\n",
              "      <td>3112.855969</td>\n",
              "      <td>9.870732</td>\n",
              "      <td>8.580358</td>\n",
              "      <td>15.915928</td>\n",
              "      <td>6.101585</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>128.000000</td>\n",
              "      <td>126.000000</td>\n",
              "      <td>3.000000</td>\n",
              "      <td>5047.000000</td>\n",
              "      <td>23.570000</td>\n",
              "      <td>5504.000000</td>\n",
              "      <td>60.570000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>23.000000</td>\n",
              "      <td>2.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>189.500000</td>\n",
              "      <td>179.000000</td>\n",
              "      <td>16.000000</td>\n",
              "      <td>5760.000000</td>\n",
              "      <td>35.017500</td>\n",
              "      <td>7422.250000</td>\n",
              "      <td>74.125000</td>\n",
              "      <td>6.000000</td>\n",
              "      <td>35.000000</td>\n",
              "      <td>8.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>229.000000</td>\n",
              "      <td>217.500000</td>\n",
              "      <td>29.500000</td>\n",
              "      <td>6855.000000</td>\n",
              "      <td>38.730000</td>\n",
              "      <td>9144.500000</td>\n",
              "      <td>78.960000</td>\n",
              "      <td>10.500000</td>\n",
              "      <td>41.500000</td>\n",
              "      <td>13.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>274.750000</td>\n",
              "      <td>256.500000</td>\n",
              "      <td>40.000000</td>\n",
              "      <td>8661.250000</td>\n",
              "      <td>42.340000</td>\n",
              "      <td>10957.000000</td>\n",
              "      <td>87.167500</td>\n",
              "      <td>16.250000</td>\n",
              "      <td>52.250000</td>\n",
              "      <td>16.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>463.000000</td>\n",
              "      <td>452.000000</td>\n",
              "      <td>84.000000</td>\n",
              "      <td>18426.000000</td>\n",
              "      <td>58.070000</td>\n",
              "      <td>21368.000000</td>\n",
              "      <td>117.000000</td>\n",
              "      <td>49.000000</td>\n",
              "      <td>96.000000</td>\n",
              "      <td>34.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              Mat        Inns         NO          Runs        Ave  \\\n",
              "count   92.000000   92.000000  92.000000     92.000000  92.000000   \n",
              "mean   241.021739  226.891304  29.217391   7617.880435  39.145761   \n",
              "std     73.548964   69.194813  15.413585   2490.710229   6.264166   \n",
              "min    128.000000  126.000000   3.000000   5047.000000  23.570000   \n",
              "25%    189.500000  179.000000  16.000000   5760.000000  35.017500   \n",
              "50%    229.000000  217.500000  29.500000   6855.000000  38.730000   \n",
              "75%    274.750000  256.500000  40.000000   8661.250000  42.340000   \n",
              "max    463.000000  452.000000  84.000000  18426.000000  58.070000   \n",
              "\n",
              "                 BF          SR        100         50          0  \n",
              "count     92.000000   92.000000  92.000000  92.000000  92.000000  \n",
              "mean    9560.043478   80.414239  12.217391  46.043478  12.847826  \n",
              "std     3112.855969    9.870732   8.580358  15.915928   6.101585  \n",
              "min     5504.000000   60.570000   0.000000  23.000000   2.000000  \n",
              "25%     7422.250000   74.125000   6.000000  35.000000   8.000000  \n",
              "50%     9144.500000   78.960000  10.500000  41.500000  13.000000  \n",
              "75%    10957.000000   87.167500  16.250000  52.250000  16.000000  \n",
              "max    21368.000000  117.000000  49.000000  96.000000  34.000000  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# checking data statistics\n",
        "display(df.describe())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ioXEtL3ldn_0",
        "outputId": "5662e3b0-b91e-40a6-f0da-d0c38970c9cb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Index(['Player', 'Span', 'Mat', 'Inns', 'NO', 'Runs', 'HS', 'Ave', 'BF', 'SR',\n",
            "       '100', '50', '0', '4s', '6s'],\n",
            "      dtype='object')\n"
          ]
        }
      ],
      "source": [
        "# column names\n",
        "print(df.columns)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y8k5_uuIdn_3"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "Lecture_01.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}