data_cleaning/05-inconsistent-data-entry.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "05285bf0",
   "metadata": {
    "papermill": {
     "duration": 0.016005,
     "end_time": "2021-08-22T19:01:55.516254",
     "exception": false,
     "start_time": "2021-08-22T19:01:55.500249",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**This notebook is an exercise in the [Data Cleaning](https://www.kaggle.com/learn/data-cleaning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/alexisbcook/inconsistent-data-entry).**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fc27f9b",
   "metadata": {
    "papermill": {
     "duration": 0.014115,
     "end_time": "2021-08-22T19:01:55.546925",
     "exception": false,
     "start_time": "2021-08-22T19:01:55.532810",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this exercise, you'll apply what you learned in the **Inconsistent data entry** tutorial.\n",
    "\n",
    "# Setup\n",
    "\n",
    "The questions below will give you feedback on your work. Run the following cell to set up the feedback system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "734649c7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T19:01:55.584253Z",
     "iopub.status.busy": "2021-08-22T19:01:55.583702Z",
     "iopub.status.idle": "2021-08-22T19:01:55.751887Z",
     "shell.execute_reply": "2021-08-22T19:01:55.751164Z",
     "shell.execute_reply.started": "2021-08-22T18:51:45.049415Z"
    },
    "papermill": {
     "duration": 0.190493,
     "end_time": "2021-08-22T19:01:55.752038",
     "exception": false,
     "start_time": "2021-08-22T19:01:55.561545",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setup Complete\n"
     ]
    }
   ],
   "source": [
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.data_cleaning.ex5 import *\n",
    "print(\"Setup Complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6b390f4",
   "metadata": {
    "papermill": {
     "duration": 0.014648,
     "end_time": "2021-08-22T19:01:55.782048",
     "exception": false,
     "start_time": "2021-08-22T19:01:55.767400",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Get our environment set up\n",
    "\n",
    "The first thing we'll need to do is load in the libraries and dataset we'll be using.  We use the same dataset from the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2eae0c27",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T19:01:55.817402Z",
     "iopub.status.busy": "2021-08-22T19:01:55.816816Z",
     "iopub.status.idle": "2021-08-22T19:01:55.829622Z",
     "shell.execute_reply": "2021-08-22T19:01:55.830141Z",
     "shell.execute_reply.started": "2021-08-22T18:51:45.588137Z"
    },
    "papermill": {
     "duration": 0.033258,
     "end_time": "2021-08-22T19:01:55.830322",
     "exception": false,
     "start_time": "2021-08-22T19:01:55.797064",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# modules we'll use\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# helpful modules\n",
    "import fuzzywuzzy\n",
    "from fuzzywuzzy import process\n",
    "import chardet\n",
    "\n",
    "# read in all our data\n",
    "professors = pd.read_csv(\"../input/pakistan-intellectual-capital/pakistan_intellectual_capital.csv\")\n",
    "\n",
    "# set seed for reproducibility\n",
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f52709a9",
   "metadata": {
    "papermill": {
     "duration": 0.014811,
     "end_time": "2021-08-22T19:01:55.860831",
     "exception": false,
     "start_time": "2021-08-22T19:01:55.846020",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Next, we'll redo all of the work that we did in the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e72e3d31",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T19:01:55.894419Z",
     "iopub.status.busy": "2021-08-22T19:01:55.893742Z",
     "iopub.status.idle": "2021-08-22T19:01:55.908482Z",
     "shell.execute_reply": "2021-08-22T19:01:55.907843Z",
     "shell.execute_reply.started": "2021-08-22T18:51:46.779104Z"
    },
    "papermill": {
     "duration": 0.032633,
     "end_time": "2021-08-22T19:01:55.908613",
     "exception": false,
     "start_time": "2021-08-22T19:01:55.875980",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All done!\n"
     ]
    }
   ],
   "source": [
    "# convert to lower case\n",
    "professors['Country'] = professors['Country'].str.lower()\n",
    "# remove trailing white spaces\n",
    "professors['Country'] = professors['Country'].str.strip()\n",
    "\n",
    "# get the top 10 closest matches to \"south korea\"\n",
    "countries = professors['Country'].unique()\n",
    "matches = fuzzywuzzy.process.extract(\"south korea\", countries, limit=10, scorer=fuzzywuzzy.fuzz.token_sort_ratio)\n",
    "\n",
    "def replace_matches_in_column(df, column, string_to_match, min_ratio = 47):\n",
    "    # get a list of unique strings\n",
    "    strings = df[column].unique()\n",
    "    \n",
    "    # get the top 10 closest matches to our input string\n",
    "    matches = fuzzywuzzy.process.extract(string_to_match, strings, \n",
    "                                         limit=10, scorer=fuzzywuzzy.fuzz.token_sort_ratio)\n",
    "\n",
    "    # only get matches with a ratio > 90\n",
    "    close_matches = [matches[0] for matches in matches if matches[1] >= min_ratio]\n",
    "\n",
    "    # get the rows of all the close matches in our dataframe\n",
    "    rows_with_matches = df[column].isin(close_matches)\n",
    "\n",
    "    # replace all rows with close matches with the input matches \n",
    "    df.loc[rows_with_matches, column] = string_to_match\n",
    "    \n",
    "    # let us know the function's done\n",
    "    print(\"All done!\")\n",
    "    \n",
    "replace_matches_in_column(df=professors, column='Country', string_to_match=\"south korea\")\n",
    "countries = professors['Country'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec7de037",
   "metadata": {
    "papermill": {
     "duration": 0.015826,
     "end_time": "2021-08-22T19:01:55.940243",
     "exception": false,
     "start_time": "2021-08-22T19:01:55.924417",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 1) Examine another column\n",
    "\n",
    "Write code below to take a look at all the unique values in the \"Graduated from\" column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "87c5e0f0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T19:01:55.980894Z",
     "iopub.status.busy": "2021-08-22T19:01:55.980264Z",
     "iopub.status.idle": "2021-08-22T19:01:55.984712Z",
     "shell.execute_reply": "2021-08-22T19:01:55.985186Z",
     "shell.execute_reply.started": "2021-08-22T18:51:53.001074Z"
    },
    "papermill": {
     "duration": 0.029515,
     "end_time": "2021-08-22T19:01:55.985346",
     "exception": false,
     "start_time": "2021-08-22T19:01:55.955831",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Asian Institute of Technology',\n",
       "       'Balochistan University of Information Technology, Engineering and Management Sciences',\n",
       "       'University of Balochistan',\n",
       "       \"Sardar Bahadur Khan Women's University\",\n",
       "       'SRH Hochschule Heidelberg',\n",
       "       'Institute of Business Administration,Karachi', 'DUET,Karachi',\n",
       "       'University of Turbat', 'University of Vienna',\n",
       "       'Monash University', 'University of Stirling',\n",
       "       'Chinese Academy of Sciences', 'University of Innsbruck',\n",
       "       'Vienna University of Technology', 'University of Paris-Est',\n",
       "       'The University of Cambridge', 'Harbin Institute of Technology',\n",
       "       'University of Nice, Sophia Antipolis', 'The University of York',\n",
       "       'Galilée - Université Paris 13', 'University of Bedfordshire',\n",
       "       'North Dakota State University', 'Kyungpook National University',\n",
       "       'The University of Manchester',\n",
       "       'National University of Sciences and Technology',\n",
       "       'FAST– National University of Computer and Emerging Sciences',\n",
       "       'Capital University of Science & Technology', 'Gomal University',\n",
       "       'University of Malaya', 'KTH Royal Institute of Technology',\n",
       "       'University of Technology',\n",
       "       'COMSATS Institute of Information Technology',\n",
       "       'Government College University', 'Mohammad Ali Jinnah University',\n",
       "       'Shaheed Zulfikar Ali Bhutto Institute of Science and Technology',\n",
       "       'Blekinge Institute of Technology', 'University of Grenoble',\n",
       "       'Politecnico di Torino', '\\xa0University of Missouri, KC',\n",
       "       'University of Bonn', 'University of Paris',\n",
       "       'The University of Leeds', '\\xa0University of Windsor',\n",
       "       '\\xa0National University of Sciences and Technology-NIIT',\n",
       "       'University of Trento', 'Stockholm University',\n",
       "       'University of New South Wales, Sydney',\n",
       "       'Kingston University London', 'Griffith University',\n",
       "       'University of Salford', 'Loughborough University',\n",
       "       'International Islamic University,Islamabad',\n",
       "       'University of Central Missouri',\n",
       "       'Riphah International University', 'University of BedfordShire',\n",
       "       'University of Illinois', 'University Of Oslo',\n",
       "       'Nancy 2 University', 'University of Limerick',\n",
       "       'Ghulam Ishaq Khan Institute of Science and Technology',\n",
       "       'University Of Waterloo', 'University of Stuttgart',\n",
       "       'Liverpool John Moores University', 'University Of Caen',\n",
       "       'Paris Tech University of Eurecom', 'University Of Salford',\n",
       "       'Lahore University of Management Sciences',\n",
       "       'Dresden University Of Technology, Dresden\\xa0',\n",
       "       'COMSATS Institute of Information Technology,Vehari',\n",
       "       'COMSATS Institute of Information Technology,Wah Cantt',\n",
       "       'TU Berlin',\n",
       "       'FAST– National University of Computer and Emerging Sciences,Islamabad',\n",
       "       'Tsinghua University', 'The University of Auckland',\n",
       "       'IQRA University', 'Universiti Teknologi PETRONAS',\n",
       "       'COMSATS Institute of Information Technology,Islamabad',\n",
       "       'Razak School of Engineering and Advanced Technology, Universiti Teknologi Malaysia (UTM)',\n",
       "       'National University of Modern Languages',\n",
       "       'University of Engineering and Technology',\n",
       "       'University Institute of Information Technology',\n",
       "       'University of Arid Agriculture', 'Quaid-i-Azam University',\n",
       "       'Queen Mary University of London',\n",
       "       'Pakistan Institute of Engineering and Applied Sciences',\n",
       "       'Pohang University of Science and Technology',\n",
       "       'Uppsala University', 'Kyung Hee University',\n",
       "       'University of Liverpool', 'University of Sunderland',\n",
       "       'Mid Sweden University', 'Bahria University,Islamabad',\n",
       "       'Chosun University', 'University of Sussex',\n",
       "       'Paris Descartes University', 'University of Leicester',\n",
       "       'University of Porto', 'University of Manchester',\n",
       "       'Université Henri Poincaré, Nancy 1,', 'Bahria University',\n",
       "       'Centre for Advanced Studies in Engineering',\n",
       "       'Norwegian University of Science and Technology (NTNU),',\n",
       "       'The Islamia University of Bahawalpur ', 'Universiti Technologi',\n",
       "       'California State University', 'University of Genova',\n",
       "       'University of Engineering and Technology,Taxila',\n",
       "       'University of\\xa0Liverpool John Moores University', 'Guildford',\n",
       "       'University of Bradford', 'Graz University of Technology',\n",
       "       'Huazhong University of Science and Technology (HUST), Wuhan',\n",
       "       'University of Konstanz',\n",
       "       'National University of Modern Languages,Islamabad',\n",
       "       'FAST– National University of Computer and Emerging Sciences,Lahore',\n",
       "       'Gwangju Institute of Science and Technology',\n",
       "       'University of Birmingham', 'Manchester University',\n",
       "       'Beijing Institute of Technology', 'University of Paisley',\n",
       "       'Univ of Porto/Univ of Aveiro Portugal/Uni of Minho',\n",
       "       'University of Peshawar', 'Universität Salzburg',\n",
       "       'Colorado State University', 'University of Virginia',\n",
       "       'University of Orleans', 'Zhejiang University',\n",
       "       'University of Leeds', 'Foundation University',\n",
       "       'Barani Institute of Information Technology', 'Abasyn University',\n",
       "       'Pir Mehr Ali Shah Arid Agriculture University', 'Preston',\n",
       "       'University of Bergen', 'Universtiy of Lahore',\n",
       "       'HITEC University,Taxila', 'Allama Iqbal Open University',\n",
       "       'University of Wales,Aberystwyth', '\\xa0University of Bonn',\n",
       "       '\\xa0Hongik University',\n",
       "       'Skolkovo Institute of Science and Technology,\\xa0',\n",
       "       'Agricultural University Peshawar', 'National Textile University',\n",
       "       'FAST– National University of Computer and Emerging Sciences,Chiniot-Faisalabad',\n",
       "       'FAST– National University of Computer and Emerging Sciences,Peshawar',\n",
       "       '\\xa0Boston University', 'Brunel University',\n",
       "       'George Washington University', 'University of the Punjab',\n",
       "       '\\xa0University of Bedfordshire',\n",
       "       'University Of Southern California', 'Beihang University',\n",
       "       'Institute of Business Administration',\n",
       "       'Abdus Salam School of Mathematical Sciences,GC University',\n",
       "       'Linköping University',\n",
       "       'National College of Business Administration and Economics',\n",
       "       'Åbo Akademi University,', 'University of Central Punjab',\n",
       "       'University of Ulm', 'University of Agriculture',\n",
       "       'University of Notre Dame Indiana\\xa0',\n",
       "       'Punjab University College of Information Technology',\n",
       "       'Ilmenau University of Technology', ' Iowa State University',\n",
       "       ' University of Innsbruck', 'Vrije University, Amsterdam',\n",
       "       ' Columbia University', 'University of Freiburg',\n",
       "       ' Delft University of Technology',\n",
       "       ' University of Texas at Arlington (UTA)', ' University of Turin',\n",
       "       ' University of Central Florida', 'Saarland University',\n",
       "       'University of Central Florida', 'Oxford Brookes University',\n",
       "       'Information Technology University (ITU)',\n",
       "       'University of Canterbury', 'University of Patras',\n",
       "       'Middle East Technical University', 'University of Bristol',\n",
       "       'University of Southern California',\n",
       "       'Northeastern University,Boston', 'Purdue University',\n",
       "       'University of Plymouth', 'University of South Australia',\n",
       "       'Stanford University', 'Chalmers University of Technology',\n",
       "       'Massachusetts Institute of Technology',\n",
       "       'Sapienza University of Rome',\n",
       "       'Eindhoven University of Technology (TU/e)',\n",
       "       'United Nations University International Institute for Software Technology (UNU-IIST)',\n",
       "       'Georgetown University,DC', 'RWTH Aachen University',\n",
       "       'Columbia University',\n",
       "       'Rutgers State University of New Jersey, NJ',\n",
       "       'University of Florida', 'Technical University of Braunschweig',\n",
       "       'Carnegie Mellon University, Pittsburgh',\n",
       "       'The Ohio State University', 'National University of Singapore',\n",
       "       'University of British Columbia', 'University of Pittsburgh',\n",
       "       'The State University of New Jersey',\n",
       "       'The University of Texas at Austin',\n",
       "       'Imperial College, University of London',\n",
       "       'University of Colorado\\xa0', 'University of Bath',\n",
       "       'Tilburg University', 'Pompeu Fabra University Barcelona',\n",
       "       'University of Management and Technology',\n",
       "       'COMSATS Institute of Information Technology,Lahore',\n",
       "       'University of Agriculture, Faisalabad\\xa0',\n",
       "       'University of Engineering & Technology',\n",
       "       'University of Agriculture, Faisalabad',\n",
       "       'Fatima Jinnah Women University, Rawalpindi',\n",
       "       'Kohat University of Science & Technology, Kohat',\n",
       "       'Virtual University of Pakistan', 'Bahauddin Zakariya University',\n",
       "       'University of the Punjab,Gujranwala',\n",
       "       'Lahore College for Women University',\n",
       "       'Superior University, Lahore',\n",
       "       'Shaheed Zulfikar Ali Bhutto Institute of Science and Technology,Islamabad',\n",
       "       'University of South Florida', 'Politecnico di Milano',\n",
       "       'Abdul Wali Khan University, Mardan', 'University of Lahore',\n",
       "       'Minhaj University Lahore', 'Lahore Leads University',\n",
       "       'Middlesex University', 'Beijing Institute of Technology Beijing',\n",
       "       'Beaconhouse National University', 'Hamdard University',\n",
       "       'University Paris', 'Sindh University',\n",
       "       'NED University of Engineering And Technology',\n",
       "       'Staffordshire University', 'DePaul University, Chicago',\n",
       "       'University of Kent',\n",
       "       'Mehran University of Engineering & Technology',\n",
       "       'Quaid-e-Awam University of Engineering, Science & Technology',\n",
       "       'Shah Abdul Latif University, Khairpur',\n",
       "       'Sindh Agriculture University', 'Swansea',\n",
       "       'University of Shanghai for Science and Technology',\n",
       "       'Griffith University,Nathan Campus', 'University of Essex',\n",
       "       'Xiamen university', 'Wayne State University',\n",
       "       'The Queens University of Belfast', 'Sungkyunkwan University',\n",
       "       'Nanyang Tech University', \"Universite d'Evry Val d'Essonne\",\n",
       "       'Sir Syed University of Engineering and Technology',\n",
       "       'New York Institute of Technology', 'Fedral Urdu University',\n",
       "       'ISRA University', 'University of Karachi',\n",
       "       'South Asian University',\n",
       "       'Capital University of Science and Technology',\n",
       "       'University of Manchester Institute of Science and Technology',\n",
       "       'The University of Birmingham',\n",
       "       'Max Planck Institute for Computer Science',\n",
       "       'George Mason University', 'University of Southampton',\n",
       "       'Temple University', 'University of Bayreuth',\n",
       "       'Muroran Institute of Technology,Hokkaido',\n",
       "       'University of Bologna', 'International Islamic University',\n",
       "       'PAF-Karachi Institute of Economics and Technology',\n",
       "       'Institute of Business Administration,Sukkur',\n",
       "       'Myongji University', 'State University of New York System',\n",
       "       'SSindh Agriculture University', 'London University',\n",
       "       'Universiti Putra Malaysia Putra',\n",
       "       'University of Rome Tor Vergata', 'University of Mississippi\\xa0',\n",
       "       'University of Wales', 'University of Northampton',\n",
       "       'University of Abertay Dundee',\n",
       "       'Biztek Institute Of Business & Technology,Karachi',\n",
       "       'University of Surrey', 'Jinnah University for Women',\n",
       "       '\\xa0Nanyang Technological University',\n",
       "       'Tokyo Institute of Technology', 'NCSU',\n",
       "       'Usman Institute of Technology', 'Hanyang University, Ansan',\n",
       "       'BUKC', 'Universtiy of Karachi', 'Pace University, New York',\n",
       "       'INSA de Lyon, Rhone', 'University of Dundee',\n",
       "       'Illinois Institute of Technology',\n",
       "       'City University of Science and Technology',\n",
       "       'Usman Institute of Technology (Hamdard University)',\n",
       "       'University of Malaga', 'Manchester Metropolitan University',\n",
       "       'Kyushu University,Fukuoka',\n",
       "       'King Abdullah University of Science and Technology',\n",
       "       'INRIA Saclay Ile-de-France', 'Université de la Rochelle',\n",
       "       'University of South Brittany', 'Aston University, Birmingham',\n",
       "       'University of Agriculture Faisalabad',\n",
       "       'Hamburg University of Technology',\n",
       "       'Government College University, Faisalabad', 'JKU',\n",
       "       'University of Oviedo',\n",
       "       'Beijing University of Posts & Telecommunications',\n",
       "       'Government College University,Faisalabad',\n",
       "       'Nottingham Trent University', 'University of Glasgow',\n",
       "       'Coventry University',\n",
       "       'Riphah International University,Faisalabad',\n",
       "       'Australian National University, Caneberra',\n",
       "       'Swedish University of Agricultural Sciences, Uppsala',\n",
       "       'University of Gujrat', 'IQRA University,Islamabad',\n",
       "       'Jonkoping University', 'Colorado Technical University',\n",
       "       'Cranfield University', 'Technical University of Graz',\n",
       "       'University of York', 'Brock University Canada',\n",
       "       'University of Westminster', 'University of Saarland',\n",
       "       'The University of Queensland', 'University of Rochester',\n",
       "       'Islamia College University ',\n",
       "       'IBMS KP Agricultural University Peshawar',\n",
       "       'University of Kuala Lumpur', 'University of Regina', 'TU Wien',\n",
       "       'Swinburne University Of Technology',\n",
       "       'Institute Of Managment Sciences, Peshawar',\n",
       "       'Universiti Tun Hussein Onn Malaysia',\n",
       "       'Institute of Management Sciences, Peshawar',\n",
       "       'University of Huddersfield',\n",
       "       'University of Engineering and Technology,Peshawar',\n",
       "       'IQRA University,Karachi', 'John Moorse University, Liverpool',\n",
       "       'CECOS University of Information Technology and Emerging Sciences,Peshawar',\n",
       "       'University of the West Scotland', 'Concordia University,Montreal',\n",
       "       'JNU', 'Grenoble', 'Florida Atlantic University',\n",
       "       'Seoul National University'], dtype=object)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TODO: Your code here\n",
    "professors['Graduated from'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c981c117",
   "metadata": {
    "papermill": {
     "duration": 0.015695,
     "end_time": "2021-08-22T19:01:56.016838",
     "exception": false,
     "start_time": "2021-08-22T19:01:56.001143",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Do you notice any inconsistencies in the data?  Can any of the inconsistencies in the data be fixed by removing white spaces at the beginning and end of cells?\n",
    "\n",
    "Once you have answered these questions, run the code cell below to get credit for your work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4ea59d59",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T19:01:56.051103Z",
     "iopub.status.busy": "2021-08-22T19:01:56.050488Z",
     "iopub.status.idle": "2021-08-22T19:01:56.058616Z",
     "shell.execute_reply": "2021-08-22T19:01:56.057997Z",
     "shell.execute_reply.started": "2021-08-22T18:51:53.519745Z"
    },
    "papermill": {
     "duration": 0.026215,
     "end_time": "2021-08-22T19:01:56.058758",
     "exception": false,
     "start_time": "2021-08-22T19:01:56.032543",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.3333333333333333, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"1_ExamineProvince\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct:</span> \n",
       "\n",
       "There are inconsistencies that can be fixed by removing white spaces at the beginning and end of cells.  For instance, \"University of Central Florida\" and \" University of Central Florida\" both appear in the column."
      ],
      "text/plain": [
       "Correct: \n",
       "\n",
       "There are inconsistencies that can be fixed by removing white spaces at the beginning and end of cells.  For instance, \"University of Central Florida\" and \" University of Central Florida\" both appear in the column."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "q1.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "762697d0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T19:01:56.095328Z",
     "iopub.status.busy": "2021-08-22T19:01:56.094742Z",
     "iopub.status.idle": "2021-08-22T19:01:56.097433Z",
     "shell.execute_reply": "2021-08-22T19:01:56.096961Z",
     "shell.execute_reply.started": "2021-08-22T18:52:13.442644Z"
    },
    "papermill": {
     "duration": 0.022589,
     "end_time": "2021-08-22T19:01:56.097546",
     "exception": false,
     "start_time": "2021-08-22T19:01:56.074957",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Line below will give you a hint\n",
    "#q1.hint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85609466",
   "metadata": {
    "papermill": {
     "duration": 0.01639,
     "end_time": "2021-08-22T19:01:56.130509",
     "exception": false,
     "start_time": "2021-08-22T19:01:56.114119",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 2) Do some text pre-processing\n",
    "\n",
    "Convert every entry in the \"Graduated from\" column in the `professors` DataFrame to remove white spaces at the beginning and end of cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e9475b5a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T19:01:56.174310Z",
     "iopub.status.busy": "2021-08-22T19:01:56.173494Z",
     "iopub.status.idle": "2021-08-22T19:01:56.178544Z",
     "shell.execute_reply": "2021-08-22T19:01:56.179068Z",
     "shell.execute_reply.started": "2021-08-22T18:52:44.375059Z"
    },
    "papermill": {
     "duration": 0.031764,
     "end_time": "2021-08-22T19:01:56.179233",
     "exception": false,
     "start_time": "2021-08-22T19:01:56.147469",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.3333333333333333, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"2_TextProcessing\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# TODO: Your code here\n",
    "professors['Graduated from'] = professors['Graduated from'].str.strip()\n",
    "\n",
    "# Check your answer\n",
    "q2.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bb10d0d6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T19:01:56.218198Z",
     "iopub.status.busy": "2021-08-22T19:01:56.217640Z",
     "iopub.status.idle": "2021-08-22T19:01:56.219354Z",
     "shell.execute_reply": "2021-08-22T19:01:56.219934Z",
     "shell.execute_reply.started": "2021-08-22T18:53:00.952537Z"
    },
    "papermill": {
     "duration": 0.022911,
     "end_time": "2021-08-22T19:01:56.220072",
     "exception": false,
     "start_time": "2021-08-22T19:01:56.197161",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#q2.hint()\n",
    "#q2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcf0cf02",
   "metadata": {
    "papermill": {
     "duration": 0.018293,
     "end_time": "2021-08-22T19:01:56.256156",
     "exception": false,
     "start_time": "2021-08-22T19:01:56.237863",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 3) Continue working with countries\n",
    "\n",
    "In the tutorial, we focused on cleaning up inconsistencies in the \"Country\" column.  Run the code cell below to view the list of unique values that we ended with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "439f50c2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T19:01:56.293866Z",
     "iopub.status.busy": "2021-08-22T19:01:56.293272Z",
     "iopub.status.idle": "2021-08-22T19:01:56.301835Z",
     "shell.execute_reply": "2021-08-22T19:01:56.302357Z",
     "shell.execute_reply.started": "2021-08-22T18:53:44.606463Z"
    },
    "papermill": {
     "duration": 0.029336,
     "end_time": "2021-08-22T19:01:56.302524",
     "exception": false,
     "start_time": "2021-08-22T19:01:56.273188",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['australia', 'austria', 'canada', 'china', 'finland', 'france',\n",
       "       'germany', 'greece', 'hongkong', 'ireland', 'italy', 'japan',\n",
       "       'macau', 'malaysia', 'mauritius', 'netherland', 'new zealand',\n",
       "       'norway', 'pakistan', 'portugal', 'russian federation',\n",
       "       'saudi arabia', 'scotland', 'singapore', 'south korea', 'spain',\n",
       "       'sweden', 'thailand', 'turkey', 'uk', 'urbana', 'usa', 'usofa'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get all the unique values in the 'City' column\n",
    "countries = professors['Country'].unique()\n",
    "\n",
    "# sort them alphabetically and then take a closer look\n",
    "countries.sort()\n",
    "countries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb23a34d",
   "metadata": {
    "papermill": {
     "duration": 0.017158,
     "end_time": "2021-08-22T19:01:56.337415",
     "exception": false,
     "start_time": "2021-08-22T19:01:56.320257",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Take another look at the \"Country\" column and see if there's any more data cleaning we need to do.\n",
    "\n",
    "It looks like 'usa' and 'usofa' should be the same country.  Correct the \"Country\" column in the dataframe so that 'usofa' appears instead as 'usa'.\n",
    "\n",
    "**Use the most recent version of the DataFrame (with the whitespaces at the beginning and end of cells removed) from question 2.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "2a929c9f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T19:01:56.376822Z",
     "iopub.status.busy": "2021-08-22T19:01:56.375904Z",
     "iopub.status.idle": "2021-08-22T19:01:56.388245Z",
     "shell.execute_reply": "2021-08-22T19:01:56.388700Z",
     "shell.execute_reply.started": "2021-08-22T19:00:29.101089Z"
    },
    "papermill": {
     "duration": 0.033255,
     "end_time": "2021-08-22T19:01:56.388843",
     "exception": false,
     "start_time": "2021-08-22T19:01:56.355588",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('usa', 100), ('usofa', 75), ('austria', 60), ('australia', 50), ('spain', 50), ('urbana', 44), ('uk', 40), ('malaysia', 36), ('pakistan', 36), ('portugal', 36)]\n",
      "All done!\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.3333333333333333, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"3_WorkingWithCities\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# TODO: Your code here!\n",
    "matches = fuzzywuzzy.process.extract('usa', countries, limit=10, scorer=fuzzywuzzy.fuzz.token_sort_ratio)\n",
    "print(matches)\n",
    "replace_matches_in_column(df=professors, column='Country', string_to_match=\"usa\", min_ratio=64)\n",
    "\n",
    "# Check your answer\n",
    "q3.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4018e272",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T19:01:56.429140Z",
     "iopub.status.busy": "2021-08-22T19:01:56.428231Z",
     "iopub.status.idle": "2021-08-22T19:01:56.431500Z",
     "shell.execute_reply": "2021-08-22T19:01:56.431980Z",
     "shell.execute_reply.started": "2021-08-22T19:00:44.982954Z"
    },
    "papermill": {
     "duration": 0.024824,
     "end_time": "2021-08-22T19:01:56.432136",
     "exception": false,
     "start_time": "2021-08-22T19:01:56.407312",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#q3.hint()\n",
    "#q3.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec9b806d",
   "metadata": {
    "papermill": {
     "duration": 0.018681,
     "end_time": "2021-08-22T19:01:56.469430",
     "exception": false,
     "start_time": "2021-08-22T19:01:56.450749",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Congratulations!\n",
    "\n",
    "Congratulations for completing the **Data Cleaning** course on Kaggle Learn!\n",
    "\n",
    "To practice your new skills, you're encouraged to download and investigate some of [Kaggle's Datasets](https://www.kaggle.com/datasets)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f3ceea3",
   "metadata": {
    "papermill": {
     "duration": 0.01898,
     "end_time": "2021-08-22T19:01:56.508085",
     "exception": false,
     "start_time": "2021-08-22T19:01:56.489105",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/172650) to chat with other Learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 9.664477,
   "end_time": "2021-08-22T19:01:57.846853",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2021-08-22T19:01:48.182376",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}