Data_Analysis_with_Pandas.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Pandas\n",
        "\n",
        "Pandas is a powerful and flexible open-source data analysis and manipulation library for Python. It provides data structures for efficiently storing and manipulating tabular data and time series, along with tools for reading, writing, and processing data."
      ],
      "metadata": {
        "id": "CtSRt00W0edk"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q0ntGZKJw1n9"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Core Data Structures in Pandas\n",
        "Pandas provides two primary data structures:\n",
        "- Series: One-dimensional labeled array\n",
        "- DataFrame: Two-dimensional labeled data structure with columns of potentially different types\n",
        "\n",
        "Let's create some examples:"
      ],
      "metadata": {
        "id": "uUtfA3nt0mQd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Creating a Series\n",
        "s = pd.Series([1, 3, 5, np.nan, 6, 8])\n",
        "print(\"Series example:\")\n",
        "print(s)\n",
        "\n",
        "# Creating a DataFrame from a dictionary\n",
        "data = {\n",
        "    'Name': ['John', 'Anna', 'Peter', 'Linda'],\n",
        "    'Age': [28, 34, 29, 42],\n",
        "    'City': ['New York', 'Paris', 'Berlin', 'London'],\n",
        "    'Salary': [65000, 70000, 62000, 85000]\n",
        "}\n",
        "\n",
        "df = pd.DataFrame(data)\n",
        "print(\"\\nDataFrame example:\")\n",
        "print(df)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VuHSg30p0ygF",
        "outputId": "65e569f9-027a-49bd-985f-af5ac59c32b9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Series example:\n",
            "0    1.0\n",
            "1    3.0\n",
            "2    5.0\n",
            "3    NaN\n",
            "4    6.0\n",
            "5    8.0\n",
            "dtype: float64\n",
            "\n",
            "DataFrame example:\n",
            "    Name  Age      City  Salary\n",
            "0   John   28  New York   65000\n",
            "1   Anna   34     Paris   70000\n",
            "2  Peter   29    Berlin   62000\n",
            "3  Linda   42    London   85000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Loading Data from External Sources\n",
        "Pandas excels at importing data from various file formats and external sources."
      ],
      "metadata": {
        "id": "m5Du-r1X1I9S"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Creating a CSV file for demonstration\n",
        "df.to_csv('sample_data.csv', index=False)\n",
        "\n",
        "# Reading data from CSV\n",
        "df_from_csv = pd.read_csv('sample_data.csv')\n",
        "print(\"Data read from CSV:\")\n",
        "print(df_from_csv)\n",
        "\n",
        "# Other common data import methods include:\n",
        "# pd.read_excel('file.xlsx')\n",
        "# pd.read_json('file.json')\n",
        "# pd.read_sql('SELECT * FROM table', connection)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p4sMgc6q1OP1",
        "outputId": "3b43ec13-82b4-4489-fb1c-b16942c62673"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Data read from CSV:\n",
            "    Name  Age      City  Salary\n",
            "0   John   28  New York   65000\n",
            "1   Anna   34     Paris   70000\n",
            "2  Peter   29    Berlin   62000\n",
            "3  Linda   42    London   85000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Exploration and Basic Information\n",
        "Pandas provides multiple methods to understand and explore your dataset:"
      ],
      "metadata": {
        "id": "yfgIznTm1eb1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Display basic information about the DataFrame\n",
        "print(\"DataFrame info:\")\n",
        "df.info()\n",
        "\n",
        "# Get statistical summary\n",
        "print(\"\\nStatistical summary:\")\n",
        "print(df.describe())\n",
        "\n",
        "# First few rows\n",
        "print(\"\\nFirst 2 rows:\")\n",
        "print(df.head(2))\n",
        "\n",
        "# Last few rows\n",
        "print(\"\\nLast 2 rows:\")\n",
        "print(df.tail(2))\n",
        "\n",
        "# Column and row count\n",
        "print(\"\\nDataFrame shape (rows, columns):\", df.shape)\n",
        "\n",
        "# Column names\n",
        "print(\"\\nColumn names:\", df.columns.tolist())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "41FH9PCZ1hw7",
        "outputId": "146a3c7a-4996-4c87-a28b-ffc05c7b01ac"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "DataFrame info:\n",
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 4 entries, 0 to 3\n",
            "Data columns (total 4 columns):\n",
            " #   Column  Non-Null Count  Dtype \n",
            "---  ------  --------------  ----- \n",
            " 0   Name    4 non-null      object\n",
            " 1   Age     4 non-null      int64 \n",
            " 2   City    4 non-null      object\n",
            " 3   Salary  4 non-null      int64 \n",
            "dtypes: int64(2), object(2)\n",
            "memory usage: 260.0+ bytes\n",
            "\n",
            "Statistical summary:\n",
            "             Age        Salary\n",
            "count   4.000000      4.000000\n",
            "mean   33.250000  70500.000000\n",
            "std     6.396614  10214.368964\n",
            "min    28.000000  62000.000000\n",
            "25%    28.750000  64250.000000\n",
            "50%    31.500000  67500.000000\n",
            "75%    36.000000  73750.000000\n",
            "max    42.000000  85000.000000\n",
            "\n",
            "First 2 rows:\n",
            "   Name  Age      City  Salary\n",
            "0  John   28  New York   65000\n",
            "1  Anna   34     Paris   70000\n",
            "\n",
            "Last 2 rows:\n",
            "    Name  Age    City  Salary\n",
            "2  Peter   29  Berlin   62000\n",
            "3  Linda   42  London   85000\n",
            "\n",
            "DataFrame shape (rows, columns): (4, 4)\n",
            "\n",
            "Column names: ['Name', 'Age', 'City', 'Salary']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Selection and Indexing\n",
        "Pandas offers multiple ways to select and filter data:"
      ],
      "metadata": {
        "id": "5eLPLBVG1oHj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Selecting a single column\n",
        "print(\"Name column:\")\n",
        "print(df['Name'])\n",
        "\n",
        "# Selecting multiple columns\n",
        "print(\"\\nName and Age columns:\")\n",
        "print(df[['Name', 'Age']])\n",
        "\n",
        "# Selection by position (iloc)\n",
        "print(\"\\nFirst two rows, first three columns:\")\n",
        "print(df.iloc[0:2, 0:3])\n",
        "\n",
        "# Selection by label (loc)\n",
        "print(\"\\nRows where Name is 'John' or 'Anna':\")\n",
        "print(df.loc[df['Name'].isin(['John', 'Anna'])])\n",
        "\n",
        "# Boolean indexing\n",
        "print(\"\\nPeople older than 30:\")\n",
        "print(df[df['Age'] > 30])\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PU_E-HSh1pxV",
        "outputId": "0a6ab028-d8da-4653-9aee-6e843889d2ad"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Name column:\n",
            "0     John\n",
            "1     Anna\n",
            "2    Peter\n",
            "3    Linda\n",
            "Name: Name, dtype: object\n",
            "\n",
            "Name and Age columns:\n",
            "    Name  Age\n",
            "0   John   28\n",
            "1   Anna   34\n",
            "2  Peter   29\n",
            "3  Linda   42\n",
            "\n",
            "First two rows, first three columns:\n",
            "   Name  Age      City\n",
            "0  John   28  New York\n",
            "1  Anna   34     Paris\n",
            "\n",
            "Rows where Name is 'John' or 'Anna':\n",
            "   Name  Age      City  Salary\n",
            "0  John   28  New York   65000\n",
            "1  Anna   34     Paris   70000\n",
            "\n",
            "People older than 30:\n",
            "    Name  Age    City  Salary\n",
            "1   Anna   34   Paris   70000\n",
            "3  Linda   42  London   85000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Cleaning and Handling Missing Values\n",
        "Real-world data often contains missing or inconsistent values. Pandas provides tools to handle these issues:"
      ],
      "metadata": {
        "id": "M2TuFIJV1ufK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Creating a DataFrame with missing values\n",
        "data_missing = {\n",
        "    'A': [1, 2, np.nan, 4],\n",
        "    'B': [5, np.nan, np.nan, 8],\n",
        "    'C': [9, 10, 11, 12]\n",
        "}\n",
        "df_missing = pd.DataFrame(data_missing)\n",
        "print(\"DataFrame with missing values:\")\n",
        "print(df_missing)\n",
        "\n",
        "# Checking for missing values\n",
        "print(\"\\nMissing values count per column:\")\n",
        "print(df_missing.isna().sum())\n",
        "\n",
        "# Dropping rows with missing values\n",
        "print(\"\\nAfter dropping rows with any missing values:\")\n",
        "print(df_missing.dropna())\n",
        "\n",
        "# Filling missing values\n",
        "print(\"\\nFilling missing values with zero:\")\n",
        "print(df_missing.fillna(0))\n",
        "\n",
        "# Filling missing values with column mean\n",
        "print(\"\\nFilling missing values with column mean:\")\n",
        "print(df_missing.fillna(df_missing.mean()))\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O54jy-SO15cd",
        "outputId": "49ce57af-b9b4-4ff0-8c98-54348e75a0be"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "DataFrame with missing values:\n",
            "     A    B   C\n",
            "0  1.0  5.0   9\n",
            "1  2.0  NaN  10\n",
            "2  NaN  NaN  11\n",
            "3  4.0  8.0  12\n",
            "\n",
            "Missing values count per column:\n",
            "A    1\n",
            "B    2\n",
            "C    0\n",
            "dtype: int64\n",
            "\n",
            "After dropping rows with any missing values:\n",
            "     A    B   C\n",
            "0  1.0  5.0   9\n",
            "3  4.0  8.0  12\n",
            "\n",
            "Filling missing values with zero:\n",
            "     A    B   C\n",
            "0  1.0  5.0   9\n",
            "1  2.0  0.0  10\n",
            "2  0.0  0.0  11\n",
            "3  4.0  8.0  12\n",
            "\n",
            "Filling missing values with column mean:\n",
            "          A    B   C\n",
            "0  1.000000  5.0   9\n",
            "1  2.000000  6.5  10\n",
            "2  2.333333  6.5  11\n",
            "3  4.000000  8.0  12\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Transformation and Column Operations\n",
        "Let's explore various data transformation techniques:"
      ],
      "metadata": {
        "id": "-WmWILrM2BI6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Adding a new column\n",
        "df['Bonus'] = df['Salary'] * 0.1\n",
        "print(\"DataFrame with Bonus column:\")\n",
        "print(df)\n",
        "\n",
        "# Applying a function to a column\n",
        "df['Salary_after_tax'] = df['Salary'].apply(lambda x: x * 0.7)\n",
        "print(\"\\nAfter adding tax-adjusted salary:\")\n",
        "print(df)\n",
        "\n",
        "# Rename columns\n",
        "df_renamed = df.rename(columns={'Salary': 'Annual_Salary', 'City': 'Location'})\n",
        "print(\"\\nRenamed columns:\")\n",
        "print(df_renamed.head())\n",
        "\n",
        "# Change data types\n",
        "df['Age'] = df['Age'].astype(float)\n",
        "print(\"\\nAge column type after conversion:\", df['Age'].dtype)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YlvAZnhN2FsA",
        "outputId": "fee763b8-7d48-4340-df27-28cc632e2f94"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "DataFrame with Bonus column:\n",
            "    Name  Age      City  Salary   Bonus\n",
            "0   John   28  New York   65000  6500.0\n",
            "1   Anna   34     Paris   70000  7000.0\n",
            "2  Peter   29    Berlin   62000  6200.0\n",
            "3  Linda   42    London   85000  8500.0\n",
            "\n",
            "After adding tax-adjusted salary:\n",
            "    Name  Age      City  Salary   Bonus  Salary_after_tax\n",
            "0   John   28  New York   65000  6500.0           45500.0\n",
            "1   Anna   34     Paris   70000  7000.0           49000.0\n",
            "2  Peter   29    Berlin   62000  6200.0           43400.0\n",
            "3  Linda   42    London   85000  8500.0           59500.0\n",
            "\n",
            "Renamed columns:\n",
            "    Name  Age  Location  Annual_Salary   Bonus  Salary_after_tax\n",
            "0   John   28  New York          65000  6500.0           45500.0\n",
            "1   Anna   34     Paris          70000  7000.0           49000.0\n",
            "2  Peter   29    Berlin          62000  6200.0           43400.0\n",
            "3  Linda   42    London          85000  8500.0           59500.0\n",
            "\n",
            "Age column type after conversion: float64\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Grouping and Aggregation\n",
        "Grouping operations are essential for data analysis:"
      ],
      "metadata": {
        "id": "Y0aygddR2NzU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a larger dataset for demonstration\n",
        "data_large = {\n",
        "    'Department': ['HR', 'IT', 'Finance', 'HR', 'IT', 'Finance', 'HR'],\n",
        "    'Employee': ['Alice', 'Bob', 'Charlie', 'David', 'Eve', 'Frank', 'Grace'],\n",
        "    'Salary': [60000, 85000, 72000, 58000, 90000, 76000, 62000],\n",
        "    'Years': [3, 5, 2, 4, 6, 3, 2]\n",
        "}\n",
        "df_large = pd.DataFrame(data_large)\n",
        "print(\"Large dataset:\")\n",
        "print(df_large)\n",
        "\n",
        "# Group by department and calculate mean\n",
        "print(\"\\nAverage salary by department:\")\n",
        "print(df_large.groupby('Department')['Salary'].mean())\n",
        "\n",
        "# Multiple aggregations\n",
        "print(\"\\nMultiple aggregations by department:\")\n",
        "print(df_large.groupby('Department').agg({\n",
        "    'Salary': ['mean', 'min', 'max', 'count'],\n",
        "    'Years': ['mean', 'min', 'max']\n",
        "}))\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HuGbD60g2RMO",
        "outputId": "6e8016ad-06e9-4c36-958f-a843201cea46"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Large dataset:\n",
            "  Department Employee  Salary  Years\n",
            "0         HR    Alice   60000      3\n",
            "1         IT      Bob   85000      5\n",
            "2    Finance  Charlie   72000      2\n",
            "3         HR    David   58000      4\n",
            "4         IT      Eve   90000      6\n",
            "5    Finance    Frank   76000      3\n",
            "6         HR    Grace   62000      2\n",
            "\n",
            "Average salary by department:\n",
            "Department\n",
            "Finance    74000.0\n",
            "HR         60000.0\n",
            "IT         87500.0\n",
            "Name: Salary, dtype: float64\n",
            "\n",
            "Multiple aggregations by department:\n",
            "             Salary                     Years        \n",
            "               mean    min    max count  mean min max\n",
            "Department                                           \n",
            "Finance     74000.0  72000  76000     2   2.5   2   3\n",
            "HR          60000.0  58000  62000     3   3.0   2   4\n",
            "IT          87500.0  85000  90000     2   5.5   5   6\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Merging, Joining, and Concatenating DataFrames\n",
        "Combining data from multiple sources is a common operation:"
      ],
      "metadata": {
        "id": "Vk2B7QNr2Yhg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create two DataFrames to merge\n",
        "df1 = pd.DataFrame({\n",
        "    'Employee_ID': [1, 2, 3, 4],\n",
        "    'Name': ['John', 'Anna', 'Peter', 'Linda'],\n",
        "    'Department': ['HR', 'IT', 'Finance', 'Operations']\n",
        "})\n",
        "\n",
        "df2 = pd.DataFrame({\n",
        "    'Employee_ID': [2, 3, 4, 5],\n",
        "    'Salary': [70000, 62000, 85000, 45000],\n",
        "    'Years': [5, 3, 7, 2]\n",
        "})\n",
        "\n",
        "print(\"df1:\")\n",
        "print(df1)\n",
        "print(\"\\ndf2:\")\n",
        "print(df2)\n",
        "\n",
        "# Inner merge (only matching Employee_ID)\n",
        "print(\"\\nInner merge:\")\n",
        "print(pd.merge(df1, df2, on='Employee_ID', how='inner'))\n",
        "\n",
        "# Outer merge (all Employee_IDs)\n",
        "print(\"\\nOuter merge:\")\n",
        "print(pd.merge(df1, df2, on='Employee_ID', how='outer'))\n",
        "\n",
        "# Concatenating DataFrames\n",
        "print(\"\\nVertical concatenation:\")\n",
        "print(pd.concat([df1, df1], ignore_index=True))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dpV3UarL2cmD",
        "outputId": "97799342-bb31-4bdd-cf71-33066ee481aa"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "df1:\n",
            "   Employee_ID   Name  Department\n",
            "0            1   John          HR\n",
            "1            2   Anna          IT\n",
            "2            3  Peter     Finance\n",
            "3            4  Linda  Operations\n",
            "\n",
            "df2:\n",
            "   Employee_ID  Salary  Years\n",
            "0            2   70000      5\n",
            "1            3   62000      3\n",
            "2            4   85000      7\n",
            "3            5   45000      2\n",
            "\n",
            "Inner merge:\n",
            "   Employee_ID   Name  Department  Salary  Years\n",
            "0            2   Anna          IT   70000      5\n",
            "1            3  Peter     Finance   62000      3\n",
            "2            4  Linda  Operations   85000      7\n",
            "\n",
            "Outer merge:\n",
            "   Employee_ID   Name  Department   Salary  Years\n",
            "0            1   John          HR      NaN    NaN\n",
            "1            2   Anna          IT  70000.0    5.0\n",
            "2            3  Peter     Finance  62000.0    3.0\n",
            "3            4  Linda  Operations  85000.0    7.0\n",
            "4            5    NaN         NaN  45000.0    2.0\n",
            "\n",
            "Vertical concatenation:\n",
            "   Employee_ID   Name  Department\n",
            "0            1   John          HR\n",
            "1            2   Anna          IT\n",
            "2            3  Peter     Finance\n",
            "3            4  Linda  Operations\n",
            "4            1   John          HR\n",
            "5            2   Anna          IT\n",
            "6            3  Peter     Finance\n",
            "7            4  Linda  Operations\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Working with Time Series Data\n",
        "Pandas has excellent support for time series data:"
      ],
      "metadata": {
        "id": "NICaHQUi2h2Y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a date range\n",
        "dates = pd.date_range(start='2025-01-01', periods=6, freq='D')\n",
        "print(\"Date range:\")\n",
        "print(dates)\n",
        "\n",
        "# Create a DataFrame with the date range as index\n",
        "ts_df = pd.DataFrame({\n",
        "    'Value': np.random.randn(6),\n",
        "    'Volume': np.random.randint(100, 1000, size=6)\n",
        "}, index=dates)\n",
        "print(\"\\nTime series DataFrame:\")\n",
        "print(ts_df)\n",
        "\n",
        "# Resampling time series data\n",
        "print(\"\\nMonthly resampling (mean):\")\n",
        "print(ts_df.resample('ME').mean())\n",
        "\n",
        "# Date filtering\n",
        "print(\"\\nData after January 3rd, 2025:\")\n",
        "print(ts_df['2025-01-03':])\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OcXSyiqT2m7a",
        "outputId": "2d002fe7-1cc1-4614-a2b3-d7a5487e6655"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Date range:\n",
            "DatetimeIndex(['2025-01-01', '2025-01-02', '2025-01-03', '2025-01-04',\n",
            "               '2025-01-05', '2025-01-06'],\n",
            "              dtype='datetime64[ns]', freq='D')\n",
            "\n",
            "Time series DataFrame:\n",
            "               Value  Volume\n",
            "2025-01-01  0.408660     871\n",
            "2025-01-02 -0.946058     200\n",
            "2025-01-03 -1.001526     458\n",
            "2025-01-04 -0.494043     582\n",
            "2025-01-05 -0.477256     901\n",
            "2025-01-06 -0.017353     844\n",
            "\n",
            "Monthly resampling (mean):\n",
            "               Value      Volume\n",
            "2025-01-31 -0.421263  642.666667\n",
            "\n",
            "Data after January 3rd, 2025:\n",
            "               Value  Volume\n",
            "2025-01-03 -1.001526     458\n",
            "2025-01-04 -0.494043     582\n",
            "2025-01-05 -0.477256     901\n",
            "2025-01-06 -0.017353     844\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Pivoting and Reshaping Data\n",
        "Reshaping data is often needed for analysis and visualization:"
      ],
      "metadata": {
        "id": "eJpGj8qr2uR8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create sample data\n",
        "sales_data = pd.DataFrame({\n",
        "    'Date': ['2025-01-01', '2025-01-01', '2025-01-02', '2025-01-02', '2025-01-03'],\n",
        "    'Product': ['A', 'B', 'A', 'B', 'A'],\n",
        "    'Region': ['East', 'West', 'East', 'West', 'East'],\n",
        "    'Sales': [100, 200, 150, 250, 180]\n",
        "})\n",
        "print(\"Sales data:\")\n",
        "print(sales_data)\n",
        "\n",
        "# Create a pivot table\n",
        "pivot = sales_data.pivot_table(values='Sales', index='Date',\n",
        "                             columns='Product', aggfunc='sum')\n",
        "print(\"\\nPivot table (Sales by Date and Product):\")\n",
        "print(pivot)\n",
        "\n",
        "# More complex pivot table\n",
        "pivot_complex = sales_data.pivot_table(\n",
        "    values='Sales',\n",
        "    index='Date',\n",
        "    columns=['Product', 'Region'],\n",
        "    aggfunc='sum',\n",
        "    fill_value=0\n",
        ")\n",
        "print(\"\\nComplex pivot table:\")\n",
        "print(pivot_complex)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lrCCg63q2yTB",
        "outputId": "86e26399-53bd-4977-eb9c-b0613b18a719"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sales data:\n",
            "         Date Product Region  Sales\n",
            "0  2025-01-01       A   East    100\n",
            "1  2025-01-01       B   West    200\n",
            "2  2025-01-02       A   East    150\n",
            "3  2025-01-02       B   West    250\n",
            "4  2025-01-03       A   East    180\n",
            "\n",
            "Pivot table (Sales by Date and Product):\n",
            "Product         A      B\n",
            "Date                    \n",
            "2025-01-01  100.0  200.0\n",
            "2025-01-02  150.0  250.0\n",
            "2025-01-03  180.0    NaN\n",
            "\n",
            "Complex pivot table:\n",
            "Product       A    B\n",
            "Region     East West\n",
            "Date                \n",
            "2025-01-01  100  200\n",
            "2025-01-02  150  250\n",
            "2025-01-03  180    0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Working with Categorical Data\n",
        "Categorical data is common in many datasets:"
      ],
      "metadata": {
        "id": "b5pBpjP7228J"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Creating categorical data\n",
        "df_cat = pd.DataFrame({\n",
        "    'ID': range(1, 11),\n",
        "    'Rating': pd.Categorical(['Low', 'Medium', 'High', 'Medium', 'Low',\n",
        "                             'High', 'Low', 'Medium', 'Medium', 'High'],\n",
        "                            categories=['Low', 'Medium', 'High'],\n",
        "                            ordered=True)\n",
        "})\n",
        "print(\"DataFrame with categorical data:\")\n",
        "print(df_cat)\n",
        "\n",
        "# Get category frequencies\n",
        "print(\"\\nRating frequencies:\")\n",
        "print(df_cat['Rating'].value_counts())\n",
        "\n",
        "# One-hot encoding\n",
        "print(\"\\nOne-hot encoded ratings:\")\n",
        "print(pd.get_dummies(df_cat['Rating']))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PFpjRWmy28SH",
        "outputId": "67de2359-0bf7-4f40-c249-c60ac45794b8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "DataFrame with categorical data:\n",
            "   ID  Rating\n",
            "0   1     Low\n",
            "1   2  Medium\n",
            "2   3    High\n",
            "3   4  Medium\n",
            "4   5     Low\n",
            "5   6    High\n",
            "6   7     Low\n",
            "7   8  Medium\n",
            "8   9  Medium\n",
            "9  10    High\n",
            "\n",
            "Rating frequencies:\n",
            "Rating\n",
            "Medium    4\n",
            "Low       3\n",
            "High      3\n",
            "Name: count, dtype: int64\n",
            "\n",
            "One-hot encoded ratings:\n",
            "     Low  Medium   High\n",
            "0   True   False  False\n",
            "1  False    True  False\n",
            "2  False   False   True\n",
            "3  False    True  False\n",
            "4   True   False  False\n",
            "5  False   False   True\n",
            "6   True   False  False\n",
            "7  False    True  False\n",
            "8  False    True  False\n",
            "9  False   False   True\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Example: Data Analysis with Pandas\n",
        "Let's put everything together with a more comprehensive example:"
      ],
      "metadata": {
        "id": "3LqXptZm3AXR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a more realistic dataset\n",
        "np.random.seed(42)  # For reproducibility\n",
        "dates = pd.date_range('2025-01-01', periods=100, freq='D')\n",
        "departments = ['HR', 'IT', 'Finance', 'Marketing', 'Operations']\n",
        "regions = ['North', 'South', 'East', 'West']\n",
        "\n",
        "df_analysis = pd.DataFrame({\n",
        "    'Date': np.random.choice(dates, 500),\n",
        "    'Department': np.random.choice(departments, 500),\n",
        "    'Region': np.random.choice(regions, 500),\n",
        "    'Sales': np.random.randint(1000, 10000, 500),\n",
        "    'Expenses': np.random.randint(500, 5000, 500),\n",
        "    'Employees': np.random.randint(1, 50, 500)\n",
        "})\n",
        "\n",
        "# Data preparation\n",
        "df_analysis['Date'] = pd.to_datetime(df_analysis['Date'])\n",
        "df_analysis['Month'] = df_analysis['Date'].dt.month\n",
        "df_analysis['Profit'] = df_analysis['Sales'] - df_analysis['Expenses']\n",
        "df_analysis['Efficiency'] = df_analysis['Sales'] / df_analysis['Employees']\n",
        "\n",
        "print(\"Analysis dataset:\")\n",
        "print(df_analysis.head())\n",
        "\n",
        "# Basic statistics by Department\n",
        "dept_stats = df_analysis.groupby('Department').agg({\n",
        "    'Sales': ['sum', 'mean'],\n",
        "    'Expenses': ['sum', 'mean'],\n",
        "    'Profit': ['sum', 'mean'],\n",
        "    'Efficiency': 'mean'\n",
        "})\n",
        "print(\"\\nDepartment statistics:\")\n",
        "print(dept_stats)\n",
        "\n",
        "# Monthly trends\n",
        "monthly_trends = df_analysis.groupby(['Month', 'Department'])['Profit'].sum().unstack()\n",
        "print(\"\\nMonthly profit trends by department:\")\n",
        "print(monthly_trends)\n",
        "\n",
        "# Top performing regions\n",
        "region_performance = df_analysis.groupby('Region').agg({\n",
        "    'Sales': 'sum',\n",
        "    'Profit': 'sum',\n",
        "    'Efficiency': 'mean'\n",
        "}).sort_values('Profit', ascending=False)\n",
        "print(\"\\nRegion performance (ranked by profit):\")\n",
        "print(region_performance)\n",
        "\n",
        "# Correlation analysis\n",
        "correlation = df_analysis[['Sales', 'Expenses', 'Employees', 'Profit', 'Efficiency']].corr()\n",
        "print(\"\\nCorrelation matrix:\")\n",
        "print(correlation)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "euHOZ_JZ3D9M",
        "outputId": "174bbd51-2b23-494e-c169-183090853c35"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Analysis dataset:\n",
            "        Date  Department Region  Sales  Expenses  Employees  Month  Profit  \\\n",
            "0 2025-02-21   Marketing  North   5788      1739         29      2    4049   \n",
            "1 2025-04-03          IT   East   6490       625         30      4    5865   \n",
            "2 2025-01-15     Finance   East   5124      4565         39      1     559   \n",
            "3 2025-03-13  Operations   West   9340      4812         44      3    4528   \n",
            "4 2025-03-02          HR  South   4850      4368         47      3     482   \n",
            "\n",
            "   Efficiency  \n",
            "0  199.586207  \n",
            "1  216.333333  \n",
            "2  131.384615  \n",
            "3  212.272727  \n",
            "4  103.191489  \n",
            "\n",
            "Department statistics:\n",
            "             Sales              Expenses               Profit               \\\n",
            "               sum         mean      sum         mean     sum         mean   \n",
            "Department                                                                   \n",
            "Finance     477563  5365.876404   263047  2955.584270  214516  2410.292135   \n",
            "HR          569968  5533.669903   266474  2587.126214  303494  2946.543689   \n",
            "IT          497208  4874.588235   298239  2923.911765  198969  1950.676471   \n",
            "Marketing   537964  5662.778947   256459  2699.568421  281505  2963.210526   \n",
            "Operations  597728  5384.936937   319557  2878.891892  278171  2506.045045   \n",
            "\n",
            "            Efficiency  \n",
            "                  mean  \n",
            "Department              \n",
            "Finance     474.793014  \n",
            "HR          543.015337  \n",
            "IT          447.692670  \n",
            "Marketing   475.720938  \n",
            "Operations  400.729300  \n",
            "\n",
            "Monthly profit trends by department:\n",
            "Department  Finance      HR     IT  Marketing  Operations\n",
            "Month                                                    \n",
            "1             60265  101658  56633      91278      105387\n",
            "2             65865   76437  57038      56894       73005\n",
            "3             67811   83394  60391      97487       66686\n",
            "4             20575   42005  24907      35846       33093\n",
            "\n",
            "Region performance (ranked by profit):\n",
            "         Sales  Profit  Efficiency\n",
            "Region                            \n",
            "South   708374  363562  433.445252\n",
            "West    642540  329259  431.242259\n",
            "East    674318  295098  447.829293\n",
            "North   655199  288736  551.123047\n",
            "\n",
            "Correlation matrix:\n",
            "               Sales  Expenses  Employees    Profit  Efficiency\n",
            "Sales       1.000000 -0.059600   0.006934  0.899572    0.288226\n",
            "Expenses   -0.059600  1.000000   0.035586 -0.489611   -0.109851\n",
            "Employees   0.006934  0.035586   1.000000 -0.009514   -0.525680\n",
            "Profit      0.899572 -0.489611  -0.009514  1.000000    0.299829\n",
            "Efficiency  0.288226 -0.109851  -0.525680  0.299829    1.000000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Pandas is an indispensable tool for data science, providing capabilities for data manipulation, cleaning, analysis, and exploration. The methods and techniques demonstrated in this notebook represent just a subset of Pandas' functionality covering most common operations needed for data science."
      ],
      "metadata": {
        "id": "S_q5Eiby3PDq"
      }
    }
  ]
}