data_cleaning/04-character-encodings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f3a14e62",
   "metadata": {
    "papermill": {
     "duration": 0.015566,
     "end_time": "2021-08-22T18:02:06.735418",
     "exception": false,
     "start_time": "2021-08-22T18:02:06.719852",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**This notebook is an exercise in the [Data Cleaning](https://www.kaggle.com/learn/data-cleaning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/alexisbcook/character-encodings).**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf6486a4",
   "metadata": {
    "papermill": {
     "duration": 0.014071,
     "end_time": "2021-08-22T18:02:06.764676",
     "exception": false,
     "start_time": "2021-08-22T18:02:06.750605",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this exercise, you'll apply what you learned in the **Character encodings** tutorial.\n",
    "\n",
    "# Setup\n",
    "\n",
    "The questions below will give you feedback on your work. Run the following cell to set up the feedback system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "35e0998e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T18:02:06.798825Z",
     "iopub.status.busy": "2021-08-22T18:02:06.797548Z",
     "iopub.status.idle": "2021-08-22T18:02:07.042717Z",
     "shell.execute_reply": "2021-08-22T18:02:07.043299Z",
     "shell.execute_reply.started": "2021-08-22T17:39:22.821325Z"
    },
    "papermill": {
     "duration": 0.264367,
     "end_time": "2021-08-22T18:02:07.043675",
     "exception": false,
     "start_time": "2021-08-22T18:02:06.779308",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setup Complete\n"
     ]
    }
   ],
   "source": [
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.data_cleaning.ex4 import *\n",
    "print(\"Setup Complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da3f5241",
   "metadata": {
    "papermill": {
     "duration": 0.014483,
     "end_time": "2021-08-22T18:02:07.073990",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.059507",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Get our environment set up\n",
    "\n",
    "The first thing we'll need to do is load in the libraries we'll be using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9164e7ec",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T18:02:07.111114Z",
     "iopub.status.busy": "2021-08-22T18:02:07.110351Z",
     "iopub.status.idle": "2021-08-22T18:02:07.112205Z",
     "shell.execute_reply": "2021-08-22T18:02:07.112827Z",
     "shell.execute_reply.started": "2021-08-22T17:39:22.984446Z"
    },
    "papermill": {
     "duration": 0.024182,
     "end_time": "2021-08-22T18:02:07.113052",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.088870",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# modules we'll use\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# helpful character encoding module\n",
    "import chardet\n",
    "\n",
    "# set seed for reproducibility\n",
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f32e290",
   "metadata": {
    "papermill": {
     "duration": 0.014702,
     "end_time": "2021-08-22T18:02:07.142789",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.128087",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 1) What are encodings?\n",
    "\n",
    "You're working with a dataset composed of bytes.  Run the code cell below to print a sample entry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5c6df216",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T18:02:07.176181Z",
     "iopub.status.busy": "2021-08-22T18:02:07.175435Z",
     "iopub.status.idle": "2021-08-22T18:02:07.181978Z",
     "shell.execute_reply": "2021-08-22T18:02:07.181289Z",
     "shell.execute_reply.started": "2021-08-22T17:39:22.991696Z"
    },
    "papermill": {
     "duration": 0.024585,
     "end_time": "2021-08-22T18:02:07.182278",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.157693",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'\\xa7A\\xa6n'\n",
      "data type: <class 'bytes'>\n"
     ]
    }
   ],
   "source": [
    "sample_entry = b'\\xa7A\\xa6n'\n",
    "print(sample_entry)\n",
    "print('data type:', type(sample_entry))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fbdab63",
   "metadata": {
    "papermill": {
     "duration": 0.014549,
     "end_time": "2021-08-22T18:02:07.212672",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.198123",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You notice that it doesn't use the standard UTF-8 encoding. \n",
    "\n",
    "Use the next code cell to create a variable `new_entry` that changes the encoding from `\"big5-tw\"` to `\"utf-8\"`.  `new_entry` should have the bytes datatype."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d9fe3463",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T18:02:07.253599Z",
     "iopub.status.busy": "2021-08-22T18:02:07.250186Z",
     "iopub.status.idle": "2021-08-22T18:02:07.258747Z",
     "shell.execute_reply": "2021-08-22T18:02:07.259209Z",
     "shell.execute_reply.started": "2021-08-22T17:43:28.970218Z"
    },
    "papermill": {
     "duration": 0.03011,
     "end_time": "2021-08-22T18:02:07.259410",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.229300",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.3333333333333333, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"1_EncodingsIntro\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "new_entry = sample_entry.decode('big5-tw').encode()\n",
    "\n",
    "# Check your answer\n",
    "q1.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bad25486",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T18:02:07.296891Z",
     "iopub.status.busy": "2021-08-22T18:02:07.295931Z",
     "iopub.status.idle": "2021-08-22T18:02:07.299697Z",
     "shell.execute_reply": "2021-08-22T18:02:07.299110Z",
     "shell.execute_reply.started": "2021-08-22T17:43:31.121905Z"
    },
    "papermill": {
     "duration": 0.024342,
     "end_time": "2021-08-22T18:02:07.299873",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.275531",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#q1.hint()\n",
    "#q1.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05d87795",
   "metadata": {
    "papermill": {
     "duration": 0.015854,
     "end_time": "2021-08-22T18:02:07.331876",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.316022",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 2) Reading in files with encoding problems\n",
    "\n",
    "Use the code cell below to read in this file at path `\"../input/fatal-police-shootings-in-the-us/PoliceKillingsUS.csv\"`.  \n",
    "\n",
    "Figure out what the correct encoding should be and read in the file to a DataFrame `police_killings`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6777dba5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T18:02:07.370650Z",
     "iopub.status.busy": "2021-08-22T18:02:07.369481Z",
     "iopub.status.idle": "2021-08-22T18:02:07.398416Z",
     "shell.execute_reply": "2021-08-22T18:02:07.398983Z",
     "shell.execute_reply.started": "2021-08-22T17:45:44.153021Z"
    },
    "papermill": {
     "duration": 0.051066,
     "end_time": "2021-08-22T18:02:07.399207",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.348141",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.3333333333333333, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"2_ReadIn\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# TODO: Load in the DataFrame correctly.\n",
    "police_killings = pd.read_csv('../input/fatal-police-shootings-in-the-us/PoliceKillingsUS.csv')\n",
    "\n",
    "# Check your answer\n",
    "q2.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d2dee86",
   "metadata": {
    "papermill": {
     "duration": 0.018201,
     "end_time": "2021-08-22T18:02:07.435712",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.417511",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Feel free to use any additional code cells for supplemental work.  To get credit for finishing this question, you'll need to run `q2.check()` and get a result of **Correct**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6bf2c491",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T18:02:07.475834Z",
     "iopub.status.busy": "2021-08-22T18:02:07.475081Z",
     "iopub.status.idle": "2021-08-22T18:02:07.537868Z",
     "shell.execute_reply": "2021-08-22T18:02:07.537274Z",
     "shell.execute_reply.started": "2021-08-22T18:00:49.301434Z"
    },
    "papermill": {
     "duration": 0.083663,
     "end_time": "2021-08-22T18:02:07.538055",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.454392",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'encoding': 'ascii', 'confidence': 1.0, 'language': ''} PercentagePeopleBelowPovertyLevel.csv\n",
      "{'encoding': 'ascii', 'confidence': 1.0, 'language': ''} PoliceKillingsUS.csv\n",
      "{'encoding': 'ascii', 'confidence': 1.0, 'language': ''} MedianHouseholdIncome2015.csv\n",
      "{'encoding': 'ascii', 'confidence': 1.0, 'language': ''} PercentOver25CompletedHighSchool.csv\n",
      "{'encoding': 'ascii', 'confidence': 1.0, 'language': ''} ShareRaceByCity.csv\n"
     ]
    }
   ],
   "source": [
    "# (Optional) Use this code cell for any additional work.\n",
    "import os\n",
    "path = '../input/fatal-police-shootings-in-the-us'\n",
    "\n",
    "for file in os.listdir(path):\n",
    "    # look at the first ten thousand bytes to guess the character encoding\n",
    "    with open(os.path.join(path, file), 'rb') as rawdata:\n",
    "        print(chardet.detect(rawdata.read(10000)), file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "93c9b8d3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T18:02:07.578609Z",
     "iopub.status.busy": "2021-08-22T18:02:07.577759Z",
     "iopub.status.idle": "2021-08-22T18:02:07.581206Z",
     "shell.execute_reply": "2021-08-22T18:02:07.580613Z",
     "shell.execute_reply.started": "2021-08-22T17:50:59.580463Z"
    },
    "papermill": {
     "duration": 0.025273,
     "end_time": "2021-08-22T18:02:07.581366",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.556093",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#q2.hint()\n",
    "#q2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff56d785",
   "metadata": {
    "papermill": {
     "duration": 0.017096,
     "end_time": "2021-08-22T18:02:07.616311",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.599215",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 3) Saving your files with UTF-8 encoding\n",
    "\n",
    "Save a version of the police killings dataset to CSV with UTF-8 encoding.  Your answer will be marked correct after saving this file.  \n",
    "\n",
    "Note: When using the `to_csv()` method, supply only the name of the file (e.g., `\"my_file.csv\"`).  This saves the file at the filepath `\"/kaggle/working/my_file.csv\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0b08b5cb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T18:02:07.658438Z",
     "iopub.status.busy": "2021-08-22T18:02:07.657637Z",
     "iopub.status.idle": "2021-08-22T18:02:07.705664Z",
     "shell.execute_reply": "2021-08-22T18:02:07.706174Z",
     "shell.execute_reply.started": "2021-08-22T17:48:49.700701Z"
    },
    "papermill": {
     "duration": 0.072248,
     "end_time": "2021-08-22T18:02:07.706477",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.634229",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.3333333333333333, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"3_SaveCSV\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# TODO: Save the police killings dataset to CSV\n",
    "police_killings.to_csv('my_file.csv')\n",
    "\n",
    "# Check your answer\n",
    "q3.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "38ba9e8e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T18:02:07.750249Z",
     "iopub.status.busy": "2021-08-22T18:02:07.749561Z",
     "iopub.status.idle": "2021-08-22T18:02:07.752764Z",
     "shell.execute_reply": "2021-08-22T18:02:07.752140Z"
    },
    "papermill": {
     "duration": 0.027249,
     "end_time": "2021-08-22T18:02:07.752908",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.725659",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#q3.hint()\n",
    "#q3.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23314528",
   "metadata": {
    "papermill": {
     "duration": 0.017605,
     "end_time": "2021-08-22T18:02:07.790154",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.772549",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# (Optional) More practice\n",
    "\n",
    "Check out [this dataset of files in different character encodings](https://www.kaggle.com/rtatman/character-encoding-examples). Can you read in all the files with their original encodings and them save them out as UTF-8 files?\n",
    "\n",
    "If you have a file that's in UTF-8 but has just a couple of weird-looking characters in it, you can try out the [ftfy module](https://ftfy.readthedocs.io/en/latest/#) and see if it helps. \n",
    "\n",
    "# Keep going\n",
    "\n",
    "In the final lesson, learn how to [**clean up inconsistent text entries**](https://www.kaggle.com/alexisbcook/inconsistent-data-entry) in your dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5335333d",
   "metadata": {
    "papermill": {
     "duration": 0.017708,
     "end_time": "2021-08-22T18:02:07.826085",
     "exception": false,
     "start_time": "2021-08-22T18:02:07.808377",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/172650) to chat with other Learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 10.784599,
   "end_time": "2021-08-22T18:02:08.554375",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2021-08-22T18:01:57.769776",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}