Course 4 Waze project lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DtNBZFHO3M7n"
   },
   "source": [
    "# **Waze Project**\n",
    "**Course 4 - The Power of Statistics**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WVeXS7BNzX6e"
   },
   "source": [
    "Your team is nearing the midpoint of their user churn project. So far, you’ve completed a project proposal, and used Python to explore and analyze Waze’s user data. You’ve also used Python to create data visualizations. The next step is to use statistical methods to analyze and interpret your data.\n",
    "\n",
    "You receive a new email from Sylvester Esperanza, your project manager. Sylvester tells your team about a new request from leadership: to analyze the relationship between mean amount of rides and device type. You also discover follow-up emails from three other team members: May Santner, Chidi Ga, and Harriet Hadzic. These emails discuss the details of the analysis. They would like a statistical analysis of ride data based on device type. In particular, leadership wants to know if there is a statistically significant difference in mean amount of rides between iPhone® users and Android™ users. A final email from Chidi includes your specific assignment: to conduct a two-sample hypothesis test (t-test) to analyze the difference in the mean amount of rides between iPhone users and Android users.\n",
    "\n",
    "A notebook was structured and prepared to help you in this project. Please complete the following questions and prepare an executive summary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rgSbVJvomcVa"
   },
   "source": [
    "# **Course 4 End-of-course project: Data exploration and hypothesis testing**\n",
    "\n",
    "In this activity, you will explore the data provided and conduct a hypothesis test.\n",
    "<br/>\n",
    "\n",
    "**The purpose** of this project is to demostrate knowledge of how to conduct a two-sample hypothesis test.\n",
    "\n",
    "**The goal** is to apply descriptive statistics and hypothesis testing in Python.\n",
    "<br/>\n",
    "\n",
    "*This activity has three parts:*\n",
    "\n",
    "**Part 1:** Imports and data loading\n",
    "* What data packages will be necessary for hypothesis testing?\n",
    "\n",
    "**Part 2:** Conduct hypothesis testing\n",
    "* How did computing descriptive statistics help you analyze your data?\n",
    "\n",
    "* How did you formulate your null hypothesis and alternative hypothesis?\n",
    "\n",
    "**Part 3:** Communicate insights with stakeholders\n",
    "\n",
    "* What key business insight(s) emerged from your hypothesis test?\n",
    "\n",
    "* What business recommendations do you propose based on your results?\n",
    "\n",
    "<br/>\n",
    "\n",
    "\n",
    "Follow the instructions and answer the questions below to complete the activity. Then, you will complete an Executive Summary using the questions listed on the PACE Strategy Document.\n",
    "\n",
    "Be sure to complete this activity before moving on. The next course item will provide you with a completed exemplar to compare to your own work.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fs2A_FgMAQhe"
   },
   "source": [
    "# **Data exploration and hypothesis testing**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N8qrn_UAzX6j"
   },
   "source": [
    "<img src=\"images/Pace.png\" width=\"100\" height=\"100\" align=left>\n",
    "\n",
    "# **PACE stages**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uaTT5L94hh5s"
   },
   "source": [
    "Throughout these project notebooks, you'll see references to the problem-solving framework PACE. The following notebook components are labeled with the respective PACE stage: Plan, Analyze, Construct, and Execute."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Bb1HNu3FzX6k"
   },
   "source": [
    "<img src=\"images/Plan.png\" width=\"100\" height=\"100\" align=left>\n",
    "\n",
    "\n",
    "## **PACE: Plan**\n",
    "\n",
    "Consider the questions in your PACE Strategy Document and those below to craft your response:\n",
    "1. What is your research question for this data project? Later on, you will need to formulate the null and alternative hypotheses as the first step of your hypothesis test. Consider your research question now, at the start of this task.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "F6_h60RuXQIz"
   },
   "source": [
    "**Research question:**\n",
    "\n",
    "\"Do drivers who open the application using an iPhone have the same number of drives on average as drivers who use Android devices?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Kp3uDhhSg_qu"
   },
   "source": [
    "*Complete the following tasks to perform statistical analysis of your data:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xgQEdIN7f1Jq"
   },
   "source": [
    "### **Task 1. Imports and data loading**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ehehvUzFj0-e"
   },
   "source": [
    "Import packages and libraries needed to compute descriptive statistics and conduct a hypothesis test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e6cryW4WgiIL"
   },
   "source": [
    "<details>\n",
    "  <summary><h4><strong>Hint:</strong></h4></summary>\n",
    "\n",
    "Before you begin, recall the following Python packages and functions:\n",
    "\n",
    "*Main functions*: stats.ttest_ind(a, b, equal_var)\n",
    "\n",
    "*Other functions*: mean()\n",
    "\n",
    "*Packages*: pandas, stats.scipy\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "1Q-GL2UrgWWG"
   },
   "outputs": [],
   "source": [
    "# Import any relevant packages or libraries\n",
    "import pandas as pd\n",
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "stMe9uT3kOmF"
   },
   "source": [
    "Import the dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "2qAt6Upaeg6U"
   },
   "outputs": [],
   "source": [
    "# Load dataset into dataframe\n",
    "df = pd.read_csv('waze_dataset.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c8paWsdKzX6o"
   },
   "source": [
    "<img src=\"images/Analyze.png\" width=\"100\" height=\"100\" align=left>\n",
    "\n",
    "<img src=\"images/Construct.png\" width=\"100\" height=\"100\" align=left>\n",
    "\n",
    "## **PACE: Analyze and Construct**\n",
    "\n",
    "Consider the questions in your PACE Strategy Document and those below to craft your response:\n",
    "1. Data professionals use descriptive statistics for exploratory data analysis (EDA). How can computing descriptive statistics help you learn more about your data in this stage of your analysis?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rX5xJnWUXxsW"
   },
   "source": [
    "**Answer:**\n",
    "\n",
    "In general, descriptive statistics are useful because they let you quickly explore and understand large amounts of data. In this case, computing descriptive statistics helps you quickly compare the average amount of drives by device type.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UKhYTdVhf5B2"
   },
   "source": [
    "### **Task 2. Data exploration**\n",
    "\n",
    "Use descriptive statistics to conduct exploratory eata analysis (EDA).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VrOZH-EGrnkC"
   },
   "source": [
    "<details>\n",
    "  <summary><h4><strong>Hint:</strong></h4></summary>\n",
    "\n",
    "Refer back to *Self Review Descriptive Statistics* for this step-by-step process.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JAq7ag0pTNgl"
   },
   "source": [
    "**Note:** In the dataset, `device` is a categorical variable with the labels `iPhone` and `Android`.\n",
    "\n",
    "In order to perform this analysis, you must turn each label into an integer.  The following code assigns a `1` for an `iPhone` user and a `2` for `Android`.  It assigns this label back to the variable `device_type`.\n",
    "\n",
    "**Note:** Creating a new variable is ideal so that you don't overwrite original data.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n3_h-Gt-g7Vc"
   },
   "source": [
    "1. Create a dictionary called `map_dictionary` that contains the class labels (`'Android'` and `'iPhone'`) for keys and the values you want to convert them to (`2` and `1`) as values.\n",
    "\n",
    "2. Create a new column called `device_type` that is a copy of the `device` column.\n",
    "\n",
    "3. Use the [`map()`](https://pandas.pydata.org/docs/reference/api/pandas.Series.map.html#pandas-series-map) method on the `device_type` series. Pass `map_dictionary` as its argument. Reassign the result back to the `device_type` series.\n",
    "</br></br>\n",
    "When you pass a dictionary to the `Series.map()` method, it will replace the data in the series where that data matches the dictionary's keys. The values that get imputed are the values of the dictionary.\n",
    "\n",
    "```\n",
    "Example:\n",
    "df['column']\n",
    "```\n",
    "\n",
    "|column |\n",
    "|  :-:       |\n",
    "| A     |\n",
    "| B     |\n",
    "| A     |\n",
    "| B     |\n",
    "\n",
    "```\n",
    "map_dictionary = {'A': 2, 'B': 1}\n",
    "df['column'] = df['column'].map(map_dictionary)\n",
    "df['column']\n",
    "```\n",
    "\n",
    "|column |\n",
    "|  :-: |\n",
    "| 2    |\n",
    "| 1    |\n",
    "| 2    |\n",
    "| 1    |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 235,
     "status": "ok",
     "timestamp": 1685491918467,
     "user": {
      "displayName": "Matteo Riotto",
      "userId": "15211004813978485634"
     },
     "user_tz": 420
    },
    "id": "PFV0nIWyg4AQ",
    "outputId": "3f20be40-d7e6-4a5e-dbe2-fbef002aa259"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2\n",
       "1    1\n",
       "2    2\n",
       "3    1\n",
       "4    2\n",
       "Name: device_type, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1. Create `map_dictionary`\n",
    "map_dictionary = {'Android': 2, 'iPhone': 1}\n",
    "\n",
    "# 2. Create new `device_type` column\n",
    "df['device_type'] = df['device']\n",
    "\n",
    "# 3. Map the new column to the dictionary\n",
    "df['device_type'] = df['device_type'].map(map_dictionary)\n",
    "\n",
    "df['device_type'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-fFiNg4hdGBK"
   },
   "source": [
    "You are interested in the relationship between device type and the number of drives. One approach is to look at the average number of drives for each device type. Calculate these averages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 254,
     "status": "ok",
     "timestamp": 1683577787174,
     "user": {
      "displayName": "Johnny Hong",
      "userId": "07434138067582511810"
     },
     "user_tz": 420
    },
    "id": "2LGEWrdVXgI2",
    "outputId": "9af851a4-835f-4276-9622-6de005bd756d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device_type\n",
       "1    67.859078\n",
       "2    66.231838\n",
       "Name: drives, dtype: float64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby('device_type')['drives'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z_u-zWvGdfII"
   },
   "source": [
    "Based on the averages shown, it appears that drivers who use an iPhone device to interact with the application have a higher number of drives on average. However, this difference might arise from random sampling, rather than being a true difference in the number of drives. To assess whether the difference is statistically significant, you can conduct a hypothesis test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QpFRQDaNzX6p"
   },
   "source": [
    "\n",
    "### **Task 3. Hypothesis testing**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7Mn5GJ460HUS"
   },
   "source": [
    "\n",
    "\n",
    "Your goal is to conduct a two-sample t-test. Recall the steps for conducting a hypothesis test:\n",
    "\n",
    "\n",
    "1.   State the null hypothesis and the alternative hypothesis\n",
    "2.   Choose a signficance level\n",
    "3.   Find the p-value\n",
    "4.   Reject or fail to reject the null hypothesis\n",
    "\n",
    "**Note:** This is a t-test for two independent samples. This is the appropriate test since the two groups are independent (Android users vs. iPhone users)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pw3YTF5AGfzY"
   },
   "source": [
    "Recall the difference between the null hypothesis ($H_0$) and the alternative hypothesis ($H_A$).\n",
    "\n",
    "**Question:** What are your hypotheses for this data project?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SkDeGggZYM84"
   },
   "source": [
    "\n",
    "**Hypotheses:**\n",
    "\n",
    "$H_0$: There is no difference in average number of drives between drivers who use iPhone devices and drivers who use Androids.\n",
    "\n",
    "$H_A$: There is a difference in average number of drives between drivers who use iPhone devices and drivers who use Androids."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IVHpnfvNeTUA"
   },
   "source": [
    "Next, choose 5% as the significance level and proceed with a two-sample t-test.\n",
    "\n",
    "You can use the `stats.ttest_ind()` function to perform the test.\n",
    "\n",
    "\n",
    "**Technical note**: The default for the argument `equal_var` in `stats.ttest_ind()` is `True`, which assumes population variances are equal. This equal variance assumption might not hold in practice (that is, there is no strong reason to assume that the two groups have the same variance); you can relax this assumption by setting `equal_var` to `False`, and `stats.ttest_ind()` will perform the unequal variances $t$-test (known as Welch's `t`-test). Refer to the [scipy t-test documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html) for more information.\n",
    "\n",
    "\n",
    "1. Isolate the `drives` column for iPhone users.\n",
    "2. Isolate the `drives` column for Android users.\n",
    "3. Perform the t-test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 190,
     "status": "ok",
     "timestamp": 1683577790072,
     "user": {
      "displayName": "Johnny Hong",
      "userId": "07434138067582511810"
     },
     "user_tz": 420
    },
    "id": "rdlsXg7Dhbrd",
    "outputId": "2dfaa652-6d8f-48d5-880c-156b51698f34"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ttest_indResult(statistic=1.4635232068852353, pvalue=0.1433519726802059)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1. Isolate the `drives` column for iPhone users.\n",
    "iPhone = df[df['device_type'] == 1]['drives']\n",
    "\n",
    "# 2. Isolate the `drives` column for Android users.\n",
    "Android = df[df['device_type'] == 2]['drives']\n",
    "\n",
    "# 3. Perform the t-test\n",
    "stats.ttest_ind(a=iPhone, b=Android, equal_var=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4cx01KwnTwY7"
   },
   "source": [
    "**Question:** Based on the p-value you got above, do you reject or fail to reject the null hypothesis?\n",
    "\n",
    "> *Since the p-value is larger than the chosen significance level (5%), you fail to reject the null hypothesis. You conclude that there is **not** a statistically significant difference in the average number of drives between drivers who use iPhones and drivers who use Androids.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ab4e-W3mzX6r"
   },
   "source": [
    "<img src=\"images/Execute.png\" width=\"100\" height=\"100\" align=left>\n",
    "\n",
    "## **PACE: Execute**\n",
    "\n",
    "Consider the questions in your PACE Strategy Document to reflect on the Execute stage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h1PRfiD3hcwE"
   },
   "source": [
    "### **Task 4: Communicate insights with stakeholders**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ItreFaZKhvVw"
   },
   "source": [
    "Now that you've completed your hypothesis test, the next step is to share your findings with the Waze leadership team. Consider the following question as you prepare to write your executive summary:\n",
    "\n",
    "* What business insight(s) can you draw from the result of your hypothesis test?\n",
    "\n",
    "> *The key business insight is that drivers who use iPhone devices on average have a similar number of drives as those who use Androids.*\n",
    "\n",
    "> *One potential next step is to explore what other factors influence the variation in the number of drives, and run additonal hypothesis tests to learn more about user behavior. Further, temporary changes in marketing or user interface for the Waze app may provide more data to investigate churn.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations!** You've completed this lab. However, you may not notice a green check mark next to this item on Coursera's platform. Please continue your progress regardless of the check mark. Just click on the \"save\" icon at the top of this notebook to ensure your work has been logged."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": [
    {
     "file_id": "1su07KaK4RI6MI85tMf2H19qop9LcqqDy",
     "timestamp": 1671052190913
    },
    {
     "file_id": "1f5FzGbEQi1DvhHY8u-K32sklfpgir_pK",
     "timestamp": 1669589913543
    }
   ],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}