AMT_Claim_Prediction_FJMODEL.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "authorship_tag": "ABX9TyNnSpsveC7aMwmihOfbJSZm",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/fjadidi2001/Insurance/blob/main/Insurance_TabNet_AMT_Claim_Prediction.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split, cross_val_score\n",
        "from sklearn.preprocessing import StandardScaler, OneHotEncoder, MinMaxScaler, LabelEncoder\n",
        "from sklearn.compose import ColumnTransformer\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.impute import SimpleImputer\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.svm import SVC\n",
        "from sklearn.ensemble import StackingClassifier\n",
        "from imblearn.over_sampling import SMOTE\n",
        "from imblearn.pipeline import Pipeline as ImbPipeline\n",
        "from sklearn.utils.class_weight import compute_class_weight\n",
        "from sklearn.metrics import (\n",
        "    classification_report,\n",
        "    confusion_matrix,\n",
        "    matthews_corrcoef,  # Added for Matthews Correlation\n",
        "    roc_auc_score,      # Added for AUC score\n",
        "    accuracy_score      # Added for test accuracy\n",
        ")\n",
        "from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score\n",
        "from sklearn.metrics import precision_score, recall_score, f1_score\n",
        "from xgboost import XGBRegressor\n",
        "from tensorflow.keras.models import Model\n",
        "from tensorflow.keras.layers import Input, Dense, BatchNormalization, Dropout\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "from pytorch_tabnet.tab_model import TabNetRegressor"
      ],
      "metadata": {
        "id": "VHalzFpPozlK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Step 1: Load df and Explore\n",
        "# Mount Google Drive\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "# Specify file path\n",
        "file_path = '/content/drive/My Drive/telematics_syn.csv'\n",
        "\n",
        "# Import pandas (assuming you want to use it to read the CSV)\n",
        "import pandas as pd\n",
        "\n",
        "# Read the CSV file\n",
        "df = pd.read_csv(file_path)\n",
        "\n",
        "print(df.head())\n",
        "print(df.info())\n",
        "\n",
        "# Check for missing values\n",
        "print(df.isnull().sum())\n",
        "\n",
        "# Display basic statistics\n",
        "print(df.describe())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_T2USPsHo23n",
        "outputId": "f8fd133b-0d44-466b-eda0-5c6f7fbcf486"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n",
            "   Duration  Insured.age Insured.sex  Car.age  Marital  Car.use  Credit.score  \\\n",
            "0       366           45        Male       -1  Married  Commute         609.0   \n",
            "1       182           44      Female        3  Married  Commute         575.0   \n",
            "2       184           48      Female        6  Married  Commute         847.0   \n",
            "3       183           71        Male        6  Married  Private         842.0   \n",
            "4       183           84        Male       10  Married  Private         856.0   \n",
            "\n",
            "  Region  Annual.miles.drive  Years.noclaims  ...  Left.turn.intensity10  \\\n",
            "0  Urban             6213.71              25  ...                    1.0   \n",
            "1  Urban            12427.42              20  ...                   58.0   \n",
            "2  Urban            12427.42              14  ...                    0.0   \n",
            "3  Urban             6213.71              43  ...                    0.0   \n",
            "4  Urban             6213.71              65  ...                    2.0   \n",
            "\n",
            "   Left.turn.intensity11  Left.turn.intensity12  Right.turn.intensity08  \\\n",
            "0                    0.0                    0.0                     3.0   \n",
            "1                   24.0                   11.0                  1099.0   \n",
            "2                    0.0                    0.0                     0.0   \n",
            "3                    0.0                    0.0                     0.0   \n",
            "4                    0.0                    0.0                   325.0   \n",
            "\n",
            "   Right.turn.intensity09  Right.turn.intensity10  Right.turn.intensity11  \\\n",
            "0                     1.0                     0.0                     0.0   \n",
            "1                   615.0                   219.0                   101.0   \n",
            "2                     0.0                     0.0                     0.0   \n",
            "3                     0.0                     0.0                     0.0   \n",
            "4                   111.0                    18.0                     4.0   \n",
            "\n",
            "   Right.turn.intensity12  NB_Claim    AMT_Claim  \n",
            "0                     0.0         1  5100.171753  \n",
            "1                    40.0         1   883.554840  \n",
            "2                     0.0         0     0.000000  \n",
            "3                     0.0         0     0.000000  \n",
            "4                     2.0         0     0.000000  \n",
            "\n",
            "[5 rows x 52 columns]\n",
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 100000 entries, 0 to 99999\n",
            "Data columns (total 52 columns):\n",
            " #   Column                  Non-Null Count   Dtype  \n",
            "---  ------                  --------------   -----  \n",
            " 0   Duration                100000 non-null  int64  \n",
            " 1   Insured.age             100000 non-null  int64  \n",
            " 2   Insured.sex             100000 non-null  object \n",
            " 3   Car.age                 100000 non-null  int64  \n",
            " 4   Marital                 100000 non-null  object \n",
            " 5   Car.use                 100000 non-null  object \n",
            " 6   Credit.score            100000 non-null  float64\n",
            " 7   Region                  100000 non-null  object \n",
            " 8   Annual.miles.drive      100000 non-null  float64\n",
            " 9   Years.noclaims          100000 non-null  int64  \n",
            " 10  Territory               100000 non-null  int64  \n",
            " 11  Annual.pct.driven       100000 non-null  float64\n",
            " 12  Total.miles.driven      100000 non-null  float64\n",
            " 13  Pct.drive.mon           100000 non-null  float64\n",
            " 14  Pct.drive.tue           100000 non-null  float64\n",
            " 15  Pct.drive.wed           100000 non-null  float64\n",
            " 16  Pct.drive.thr           100000 non-null  float64\n",
            " 17  Pct.drive.fri           100000 non-null  float64\n",
            " 18  Pct.drive.sat           100000 non-null  float64\n",
            " 19  Pct.drive.sun           100000 non-null  float64\n",
            " 20  Pct.drive.2hrs          100000 non-null  float64\n",
            " 21  Pct.drive.3hrs          100000 non-null  float64\n",
            " 22  Pct.drive.4hrs          100000 non-null  float64\n",
            " 23  Pct.drive.wkday         100000 non-null  float64\n",
            " 24  Pct.drive.wkend         100000 non-null  float64\n",
            " 25  Pct.drive.rush am       100000 non-null  float64\n",
            " 26  Pct.drive.rush pm       100000 non-null  float64\n",
            " 27  Avgdays.week            100000 non-null  float64\n",
            " 28  Accel.06miles           100000 non-null  float64\n",
            " 29  Accel.08miles           100000 non-null  float64\n",
            " 30  Accel.09miles           100000 non-null  float64\n",
            " 31  Accel.11miles           100000 non-null  float64\n",
            " 32  Accel.12miles           100000 non-null  float64\n",
            " 33  Accel.14miles           100000 non-null  float64\n",
            " 34  Brake.06miles           100000 non-null  float64\n",
            " 35  Brake.08miles           100000 non-null  float64\n",
            " 36  Brake.09miles           100000 non-null  float64\n",
            " 37  Brake.11miles           100000 non-null  float64\n",
            " 38  Brake.12miles           100000 non-null  float64\n",
            " 39  Brake.14miles           100000 non-null  float64\n",
            " 40  Left.turn.intensity08   100000 non-null  float64\n",
            " 41  Left.turn.intensity09   100000 non-null  float64\n",
            " 42  Left.turn.intensity10   100000 non-null  float64\n",
            " 43  Left.turn.intensity11   100000 non-null  float64\n",
            " 44  Left.turn.intensity12   100000 non-null  float64\n",
            " 45  Right.turn.intensity08  100000 non-null  float64\n",
            " 46  Right.turn.intensity09  100000 non-null  float64\n",
            " 47  Right.turn.intensity10  100000 non-null  float64\n",
            " 48  Right.turn.intensity11  100000 non-null  float64\n",
            " 49  Right.turn.intensity12  100000 non-null  float64\n",
            " 50  NB_Claim                100000 non-null  int64  \n",
            " 51  AMT_Claim               100000 non-null  float64\n",
            "dtypes: float64(42), int64(6), object(4)\n",
            "memory usage: 39.7+ MB\n",
            "None\n",
            "Duration                  0\n",
            "Insured.age               0\n",
            "Insured.sex               0\n",
            "Car.age                   0\n",
            "Marital                   0\n",
            "Car.use                   0\n",
            "Credit.score              0\n",
            "Region                    0\n",
            "Annual.miles.drive        0\n",
            "Years.noclaims            0\n",
            "Territory                 0\n",
            "Annual.pct.driven         0\n",
            "Total.miles.driven        0\n",
            "Pct.drive.mon             0\n",
            "Pct.drive.tue             0\n",
            "Pct.drive.wed             0\n",
            "Pct.drive.thr             0\n",
            "Pct.drive.fri             0\n",
            "Pct.drive.sat             0\n",
            "Pct.drive.sun             0\n",
            "Pct.drive.2hrs            0\n",
            "Pct.drive.3hrs            0\n",
            "Pct.drive.4hrs            0\n",
            "Pct.drive.wkday           0\n",
            "Pct.drive.wkend           0\n",
            "Pct.drive.rush am         0\n",
            "Pct.drive.rush pm         0\n",
            "Avgdays.week              0\n",
            "Accel.06miles             0\n",
            "Accel.08miles             0\n",
            "Accel.09miles             0\n",
            "Accel.11miles             0\n",
            "Accel.12miles             0\n",
            "Accel.14miles             0\n",
            "Brake.06miles             0\n",
            "Brake.08miles             0\n",
            "Brake.09miles             0\n",
            "Brake.11miles             0\n",
            "Brake.12miles             0\n",
            "Brake.14miles             0\n",
            "Left.turn.intensity08     0\n",
            "Left.turn.intensity09     0\n",
            "Left.turn.intensity10     0\n",
            "Left.turn.intensity11     0\n",
            "Left.turn.intensity12     0\n",
            "Right.turn.intensity08    0\n",
            "Right.turn.intensity09    0\n",
            "Right.turn.intensity10    0\n",
            "Right.turn.intensity11    0\n",
            "Right.turn.intensity12    0\n",
            "NB_Claim                  0\n",
            "AMT_Claim                 0\n",
            "dtype: int64\n",
            "            Duration    Insured.age        Car.age   Credit.score  \\\n",
            "count  100000.000000  100000.000000  100000.000000  100000.000000   \n",
            "mean      314.204060      51.378950       5.639720     800.888870   \n",
            "std        79.746222      15.467075       4.062135      83.382316   \n",
            "min        27.000000      16.000000      -2.000000     422.000000   \n",
            "25%       200.000000      39.000000       2.000000     766.000000   \n",
            "50%       365.000000      51.000000       5.000000     825.000000   \n",
            "75%       366.000000      63.000000       8.000000     856.000000   \n",
            "max       366.000000     103.000000      20.000000     900.000000   \n",
            "\n",
            "       Annual.miles.drive  Years.noclaims      Territory  Annual.pct.driven  \\\n",
            "count       100000.000000   100000.000000  100000.000000      100000.000000   \n",
            "mean          9124.122908       28.839960      56.531390           0.502294   \n",
            "std           3826.144730       16.123717      24.036518           0.299189   \n",
            "min              0.000000        0.000000      11.000000           0.002740   \n",
            "25%           6213.710000       15.000000      35.000000           0.249315   \n",
            "50%           7456.452000       29.000000      62.000000           0.490411   \n",
            "75%          12427.420000       41.000000      78.000000           0.753425   \n",
            "max          56731.172300       79.000000      91.000000           1.000000   \n",
            "\n",
            "       Total.miles.driven  Pct.drive.mon  ...  Left.turn.intensity10  \\\n",
            "count       100000.000000  100000.000000  ...          100000.000000   \n",
            "mean          4833.575303       0.139365  ...             551.574010   \n",
            "std           4545.943016       0.042807  ...           14687.929802   \n",
            "min              0.095298       0.000000  ...               0.000000   \n",
            "25%           1529.897500       0.120894  ...               0.000000   \n",
            "50%           3468.287765       0.137909  ...               3.000000   \n",
            "75%           6779.876842       0.155203  ...              30.000000   \n",
            "max          47282.603936       0.998172  ...          794380.000000   \n",
            "\n",
            "       Left.turn.intensity11  Left.turn.intensity12  Right.turn.intensity08  \\\n",
            "count          100000.000000          100000.000000           100000.000000   \n",
            "mean              487.340690             447.758420              843.461830   \n",
            "std             14198.331308           13719.790281            11630.185503   \n",
            "min                 0.000000               0.000000                0.000000   \n",
            "25%                 0.000000               0.000000               11.000000   \n",
            "50%                 1.000000               0.000000              122.000000   \n",
            "75%                 9.000000               2.000000              680.000000   \n",
            "max            793926.000000          793170.000000           841210.000000   \n",
            "\n",
            "       Right.turn.intensity09  Right.turn.intensity10  Right.turn.intensity11  \\\n",
            "count           100000.000000           100000.000000           100000.000000   \n",
            "mean               565.056100              326.654840              246.713120   \n",
            "std              10657.402935             9460.244357             8977.569994   \n",
            "min                  0.000000                0.000000                0.000000   \n",
            "25%                  3.000000                0.000000                0.000000   \n",
            "50%                 43.000000                7.000000                2.000000   \n",
            "75%                321.000000               81.000000               27.000000   \n",
            "max             841207.000000           841200.000000           841176.000000   \n",
            "\n",
            "       Right.turn.intensity12      NB_Claim      AMT_Claim  \n",
            "count           100000.000000  100000.00000  100000.000000  \n",
            "mean               198.753690       0.04494     137.602253  \n",
            "std               8585.177049       0.21813    1264.320056  \n",
            "min                  0.000000       0.00000       0.000000  \n",
            "25%                  0.000000       0.00000       0.000000  \n",
            "50%                  0.000000       0.00000       0.000000  \n",
            "75%                  9.000000       0.00000       0.000000  \n",
            "max             841144.000000       3.00000  104074.886700  \n",
            "\n",
            "[8 rows x 48 columns]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Remove samples with AMT_Claim = 0\n",
        "df = df[df['AMT_Claim'] > 0]\n",
        "\n",
        "# Data Preprocessing\n",
        "categorical_columns = ['Insured.sex', 'Marital', 'Car.use', 'Region']\n",
        "numerical_columns = [col for col in df.columns if col not in categorical_columns + ['AMT_Claim']]\n",
        "\n",
        "# Feature Engineering\n",
        "def aggregate_harsh_driving(df):\n",
        "    df['total_accel'] = df[[col for col in df.columns if col.startswith('Accel')]].sum(axis=1)\n",
        "    df['total_brake'] = df[[col for col in df.columns if col.startswith('Brake')]].sum(axis=1)\n",
        "    df['total_left_turn'] = df[[col for col in df.columns if col.startswith('Left.turn')]].sum(axis=1)\n",
        "    df['total_right_turn'] = df[[col for col in df.columns if col.startswith('Right.turn')]].sum(axis=1)\n",
        "    return df\n",
        "\n",
        "df = aggregate_harsh_driving(df)\n",
        "\n",
        "# Split the data\n",
        "X = df.drop('AMT_Claim', axis=1)\n",
        "y = df['AMT_Claim']\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
        "\n",
        "# Preprocessing\n",
        "numeric_transformer = Pipeline(steps=[\n",
        "    ('imputer', SimpleImputer(strategy='median')),\n",
        "    ('scaler', StandardScaler())\n",
        "])\n",
        "\n",
        "categorical_transformer = Pipeline(steps=[\n",
        "    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),\n",
        "    ('onehot', OneHotEncoder(handle_unknown='ignore'))\n",
        "])\n",
        "\n",
        "preprocessor = ColumnTransformer(\n",
        "    transformers=[\n",
        "        ('num', numeric_transformer, numerical_columns),\n",
        "        ('cat', categorical_transformer, categorical_columns)\n",
        "    ])\n",
        "\n",
        "# Fit the preprocessor and transform the data\n",
        "X_train_processed = preprocessor.fit_transform(X_train)\n",
        "X_test_processed = preprocessor.transform(X_test)\n",
        "\n",
        "# Get the number of features after preprocessing\n",
        "n_features = X_train_processed.shape[1]\n"
      ],
      "metadata": {
        "id": "F7iWYwAaq5cN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        " # Replace with your actual file path\n",
        "df = df[df['AMT_Claim'] > 0]  # Consider only non-zero claims\n",
        "print(f\"Number of samples after removing zero claims: {len(df)}\")\n",
        "\n",
        "# Step 2: Define features and target\n",
        "print(\"\\nStep 2: Defining features and target\")\n",
        "categorical_columns = ['Insured.sex', 'Marital', 'Car.use', 'Region']\n",
        "numerical_columns = [col for col in df.columns if col not in categorical_columns + ['AMT_Claim']]\n",
        "X = df.drop('AMT_Claim', axis=1)\n",
        "y = df['AMT_Claim']\n",
        "\n",
        "# Step 3: Split the data\n",
        "print(\"\\nStep 3: Splitting the data\")\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
        "print(f\"Training set shape: {X_train.shape}, Test set shape: {X_test.shape}\")\n",
        "\n",
        "# Step 4: Create preprocessing pipeline\n",
        "print(\"\\nStep 4: Creating preprocessing pipeline\")\n",
        "numeric_transformer = Pipeline(steps=[\n",
        "    ('imputer', SimpleImputer(strategy='median')),\n",
        "    ('scaler', StandardScaler())\n",
        "])\n",
        "\n",
        "categorical_transformer = Pipeline(steps=[\n",
        "    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),\n",
        "    ('onehot', OneHotEncoder(handle_unknown='ignore'))\n",
        "])\n",
        "\n",
        "preprocessor = ColumnTransformer(\n",
        "    transformers=[\n",
        "        ('num', numeric_transformer, numerical_columns),\n",
        "        ('cat', categorical_transformer, categorical_columns)\n",
        "    ])\n",
        "\n",
        "# Step 5: Fit preprocessor and transform data\n",
        "print(\"\\nStep 5: Fitting preprocessor and transforming data\")\n",
        "X_train_processed = preprocessor.fit_transform(X_train)\n",
        "X_test_processed = preprocessor.transform(X_test)\n",
        "print(f\"Processed training set shape: {X_train_processed.shape}\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HXDqcURnwuov",
        "outputId": "09651a9e-590f-4f6b-ffc8-b77a571f5d3f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of samples after removing zero claims: 3864\n",
            "\n",
            "Step 2: Defining features and target\n",
            "\n",
            "Step 3: Splitting the data\n",
            "Training set shape: (3091, 55), Test set shape: (773, 55)\n",
            "\n",
            "Step 4: Creating preprocessing pipeline\n",
            "\n",
            "Step 5: Fitting preprocessor and transforming data\n",
            "Processed training set shape: (3091, 61)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Step 6: Define the FJ model (DNN + TabNet + XGBoost)\n",
        "print(\"\\nStep 6: Defining the FJ model\")\n",
        "from tensorflow.keras.models import Sequential # Import Sequential for DNN\n",
        "class FJModel:\n",
        "    def __init__(self, input_dim):\n",
        "        self.dnn = self._build_dnn(input_dim)\n",
        "        self.tabnet = TabNetRegressor()\n",
        "        self.xgb = XGBRegressor(random_state=42)\n",
        "\n",
        "    def _build_dnn(self, input_dim):\n",
        "        model = Sequential([\n",
        "            Dense(64, activation='relu', input_shape=(input_dim,)),\n",
        "            BatchNormalization(),\n",
        "            Dropout(0.3),\n",
        "            Dense(32, activation='relu'),\n",
        "            BatchNormalization(),\n",
        "            Dropout(0.3),\n",
        "            Dense(1)\n",
        "        ])\n",
        "        model.compile(optimizer=Adam(), loss='mse')\n",
        "        return model\n",
        "\n",
        "    def fit(self, X, y):\n",
        "        self.dnn.fit(X, y, epochs=50, batch_size=32, verbose=0)\n",
        "        self.tabnet.fit(X.toarray() if hasattr(X, 'toarray') else X, y.values.reshape(-1, 1)) # Convert y to NumPy array before reshape\n",
        "        self.xgb.fit(X, y)\n",
        "\n",
        "    def predict(self, X):\n",
        "        dnn_pred = self.dnn.predict(X).flatten()\n",
        "        tabnet_pred = self.tabnet.predict(X.toarray() if hasattr(X, 'toarray') else X).flatten()\n",
        "        xgb_pred = self.xgb.predict(X)\n",
        "        return (dnn_pred + tabnet_pred + xgb_pred) / 3"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "44gNruvTzNMn",
        "outputId": "20b55b85-ea80-411b-bca2-8174c348207f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Step 6: Defining the FJ model\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Step 7: Initialize and train the model\n",
        "print(\"\\nStep 7: Initializing and training the model\")\n",
        "model = FJModel(X_train_processed.shape[1])\n",
        "model.fit(X_train_processed, y_train)\n",
        "\n",
        "# Step 8: Make predictions\n",
        "print(\"\\nStep 8: Making predictions\")\n",
        "y_pred = model.predict(X_test_processed)\n",
        "\n",
        "# Step 9: Calculate metrics\n",
        "print(\"\\nStep 9: Calculating metrics\")\n",
        "mse = mean_squared_error(y_test, y_pred)\n",
        "mae = mean_absolute_error(y_test, y_pred)\n",
        "r2 = r2_score(y_test, y_pred)\n",
        "\n",
        "print(f\"Mean Squared Error: {mse}\")\n",
        "print(f\"Mean Absolute Error: {mae}\")\n",
        "print(f\"R-squared Score: {r2}\")\n",
        "\n",
        "# Step 10: Feature Importance\n",
        "print(\"\\nStep 10: Calculating feature importance\")\n",
        "feature_importance = model.xgb.feature_importances_\n",
        "feature_names = preprocessor.get_feature_names_out()\n",
        "\n",
        "importance_df = pd.DataFrame({'feature': feature_names, 'importance': feature_importance})\n",
        "importance_df = importance_df.sort_values('importance', ascending=False)\n",
        "\n",
        "print(\"Top 10 Most Important Features:\")\n",
        "print(importance_df.head(10))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0B80vY4QzTi9",
        "outputId": "9d3994e2-50e6-4c96-dcc0-c113c60155c7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Step 7: Initializing and training the model\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.\n",
            "  super().__init__(activity_regularizer=activity_regularizer, **kwargs)\n",
            "/usr/local/lib/python3.10/dist-packages/pytorch_tabnet/abstract_model.py:82: UserWarning: Device used : cuda\n",
            "  warnings.warn(f\"Device used : {self.device}\")\n",
            "/usr/local/lib/python3.10/dist-packages/pytorch_tabnet/abstract_model.py:687: UserWarning: No early stopping will be performed, last training weights will be used.\n",
            "  warnings.warn(wrn_msg)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "epoch 0  | loss: 41120514.66667|  0:00:02s\n",
            "epoch 1  | loss: 41327932.66667|  0:00:02s\n",
            "epoch 2  | loss: 41649532.66667|  0:00:02s\n",
            "epoch 3  | loss: 41404052.0|  0:00:02s\n",
            "epoch 4  | loss: 41187690.66667|  0:00:02s\n",
            "epoch 5  | loss: 41538703.33333|  0:00:02s\n",
            "epoch 6  | loss: 41239380.0|  0:00:02s\n",
            "epoch 7  | loss: 41578485.33333|  0:00:02s\n",
            "epoch 8  | loss: 41531876.0|  0:00:03s\n",
            "epoch 9  | loss: 41355580.66667|  0:00:03s\n",
            "epoch 10 | loss: 41363422.66667|  0:00:03s\n",
            "epoch 11 | loss: 41089605.33333|  0:00:03s\n",
            "epoch 12 | loss: 41200174.66667|  0:00:03s\n",
            "epoch 13 | loss: 40527448.0|  0:00:03s\n",
            "epoch 14 | loss: 40843910.0|  0:00:03s\n",
            "epoch 15 | loss: 41072924.0|  0:00:03s\n",
            "epoch 16 | loss: 41073609.33333|  0:00:04s\n",
            "epoch 17 | loss: 41116813.33333|  0:00:04s\n",
            "epoch 18 | loss: 40762462.66667|  0:00:04s\n",
            "epoch 19 | loss: 41041193.33333|  0:00:04s\n",
            "epoch 20 | loss: 41019552.0|  0:00:04s\n",
            "epoch 21 | loss: 40883738.66667|  0:00:04s\n",
            "epoch 22 | loss: 40873464.0|  0:00:04s\n",
            "epoch 23 | loss: 40328282.66667|  0:00:04s\n",
            "epoch 24 | loss: 40750072.0|  0:00:05s\n",
            "epoch 25 | loss: 40340566.66667|  0:00:05s\n",
            "epoch 26 | loss: 40413588.0|  0:00:05s\n",
            "epoch 27 | loss: 40004984.0|  0:00:05s\n",
            "epoch 28 | loss: 40434258.0|  0:00:05s\n",
            "epoch 29 | loss: 40101456.0|  0:00:05s\n",
            "epoch 30 | loss: 40087989.33333|  0:00:05s\n",
            "epoch 31 | loss: 39973965.33333|  0:00:05s\n",
            "epoch 32 | loss: 39951618.66667|  0:00:06s\n",
            "epoch 33 | loss: 39883657.33333|  0:00:06s\n",
            "epoch 34 | loss: 39605277.33333|  0:00:06s\n",
            "epoch 35 | loss: 39428785.33333|  0:00:06s\n",
            "epoch 36 | loss: 39620608.0|  0:00:06s\n",
            "epoch 37 | loss: 39415885.33333|  0:00:06s\n",
            "epoch 38 | loss: 39207242.66667|  0:00:06s\n",
            "epoch 39 | loss: 39179218.0|  0:00:06s\n",
            "epoch 40 | loss: 39133944.0|  0:00:07s\n",
            "epoch 41 | loss: 38717364.0|  0:00:07s\n",
            "epoch 42 | loss: 38855164.0|  0:00:07s\n",
            "epoch 43 | loss: 38976065.33333|  0:00:07s\n",
            "epoch 44 | loss: 38820005.33333|  0:00:07s\n",
            "epoch 45 | loss: 38551719.33333|  0:00:07s\n",
            "epoch 46 | loss: 38678585.33333|  0:00:07s\n",
            "epoch 47 | loss: 38339459.33333|  0:00:07s\n",
            "epoch 48 | loss: 38476517.33333|  0:00:08s\n",
            "epoch 49 | loss: 38165508.0|  0:00:08s\n",
            "epoch 50 | loss: 38076397.33333|  0:00:08s\n",
            "epoch 51 | loss: 38099457.33333|  0:00:08s\n",
            "epoch 52 | loss: 37890060.0|  0:00:08s\n",
            "epoch 53 | loss: 37806393.33333|  0:00:08s\n",
            "epoch 54 | loss: 37512310.0|  0:00:08s\n",
            "epoch 55 | loss: 37699324.0|  0:00:08s\n",
            "epoch 56 | loss: 37335234.66667|  0:00:09s\n",
            "epoch 57 | loss: 37232580.66667|  0:00:09s\n",
            "epoch 58 | loss: 37248315.33333|  0:00:09s\n",
            "epoch 59 | loss: 37073384.0|  0:00:09s\n",
            "epoch 60 | loss: 36736661.33333|  0:00:09s\n",
            "epoch 61 | loss: 36918645.33333|  0:00:09s\n",
            "epoch 62 | loss: 36661162.66667|  0:00:09s\n",
            "epoch 63 | loss: 36566748.0|  0:00:09s\n",
            "epoch 64 | loss: 36362750.66667|  0:00:10s\n",
            "epoch 65 | loss: 36353091.33333|  0:00:10s\n",
            "epoch 66 | loss: 36357757.33333|  0:00:10s\n",
            "epoch 67 | loss: 36031358.66667|  0:00:10s\n",
            "epoch 68 | loss: 36089762.66667|  0:00:10s\n",
            "epoch 69 | loss: 35890098.66667|  0:00:10s\n",
            "epoch 70 | loss: 35748976.0|  0:00:10s\n",
            "epoch 71 | loss: 35317184.0|  0:00:10s\n",
            "epoch 72 | loss: 35397855.33333|  0:00:11s\n",
            "epoch 73 | loss: 35436409.33333|  0:00:11s\n",
            "epoch 74 | loss: 35130651.33333|  0:00:11s\n",
            "epoch 75 | loss: 34190185.33333|  0:00:11s\n",
            "epoch 76 | loss: 34894086.66667|  0:00:12s\n",
            "epoch 77 | loss: 34678905.33333|  0:00:12s\n",
            "epoch 78 | loss: 34690746.66667|  0:00:12s\n",
            "epoch 79 | loss: 34608270.66667|  0:00:12s\n",
            "epoch 80 | loss: 34556958.0|  0:00:12s\n",
            "epoch 81 | loss: 34307061.33333|  0:00:13s\n",
            "epoch 82 | loss: 34141591.33333|  0:00:13s\n",
            "epoch 83 | loss: 33988376.66667|  0:00:13s\n",
            "epoch 84 | loss: 34141312.0|  0:00:13s\n",
            "epoch 85 | loss: 34039378.66667|  0:00:13s\n",
            "epoch 86 | loss: 33737819.33333|  0:00:14s\n",
            "epoch 87 | loss: 33674868.0|  0:00:14s\n",
            "epoch 88 | loss: 33219380.66667|  0:00:14s\n",
            "epoch 89 | loss: 32814236.66667|  0:00:14s\n",
            "epoch 90 | loss: 32966169.33333|  0:00:14s\n",
            "epoch 91 | loss: 33226816.0|  0:00:15s\n",
            "epoch 92 | loss: 33006414.66667|  0:00:15s\n",
            "epoch 93 | loss: 32838466.66667|  0:00:15s\n",
            "epoch 94 | loss: 32903798.0|  0:00:15s\n",
            "epoch 95 | loss: 32629425.33333|  0:00:15s\n",
            "epoch 96 | loss: 32398532.66667|  0:00:15s\n",
            "epoch 97 | loss: 32203781.33333|  0:00:15s\n",
            "epoch 98 | loss: 32387040.0|  0:00:15s\n",
            "epoch 99 | loss: 32143520.0|  0:00:16s\n",
            "\n",
            "Step 8: Making predictions\n",
            "\u001b[1m25/25\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 17ms/step\n",
            "\n",
            "Step 9: Calculating metrics\n",
            "Mean Squared Error: 24361113.417124562\n",
            "Mean Absolute Error: 2391.0563518193935\n",
            "R-squared Score: 0.19408436645450278\n",
            "\n",
            "Step 10: Calculating feature importance\n",
            "Top 10 Most Important Features:\n",
            "                    feature  importance\n",
            "32       num__Brake.09miles    0.146238\n",
            "46            num__NB_Claim    0.105440\n",
            "14       num__Pct.drive.sat    0.029184\n",
            "31       num__Brake.08miles    0.028139\n",
            "5       num__Years.noclaims    0.026861\n",
            "27       num__Accel.11miles    0.026561\n",
            "25       num__Accel.08miles    0.024923\n",
            "3         num__Credit.score    0.023697\n",
            "51  cat__Insured.sex_Female    0.023214\n",
            "7    num__Annual.pct.driven    0.022931\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Abstract\n",
        "The document outlines the implementation of a hybrid model called FJModel, which combines Deep Neural Networks (DNN), TabNet, and XGBoost for predicting insurance claims. The FJModel is designed to enhance prediction accuracy by training each component on the input data and averaging the predictions. It provides a step-by-step approach to defining and fitting the model using TensorFlow and XGBoost libraries while detailing the architecture of the DNN segment.\n",
        "\n",
        "### Key Points\n",
        "- The FJModel integrates three advanced machine learning techniques: DNN, TabNet, and XGBoost, to improve claim prediction accuracy.\n",
        "- The DNN architecture includes layers with dropout and batch normalization to enhance the model's generalization capability.\n",
        "- The model utilizes Adam optimizer and mean squared error as the loss function for training the DNN component.\n",
        "- TabNetRegressor is employed for its capacity to handle tabular data more effectively than conventional methods.\n",
        "- XGBoost is included for its robustness in handling various data types and achieving high performance.\n",
        "- The fit method trains all three models sequentially, ensuring that predictions can leverage the strengths of each technique.\n",
        "- The predict method computes the final prediction by averaging the outputs from the DNN, TabNet, and XGBoost models.\n",
        "\n",
        "\n",
        "The effectiveness of the FJModel (which combines DNN, TabNet, and XGBoost) in predicting insurance claims can be assessed based on several factors:\n",
        "\n",
        "1. **Model Architecture**: The DNN component uses a well-structured architecture with dropout and batch normalization, which helps prevent overfitting and improves generalization. The inclusion of TabNet and XGBoost leverages their strengths in handling tabular data and boosting performance, respectively.\n",
        "\n",
        "2. **Training Process**: The model is trained on the input data using a combination of techniques, which allows it to learn from different perspectives. The averaging of predictions from the three models can lead to more robust results compared to using a single model.\n",
        "\n",
        "3. **Loss Function**: The use of mean squared error (MSE) as the loss function for the DNN indicates that the model is focused on minimizing prediction errors, which is crucial for regression tasks like insurance claims prediction.\n",
        "\n",
        "4. **Performance Metrics**: While the code does not provide specific performance metrics (like RMSE, MAE, or R²), the effectiveness of the model can ultimately be evaluated through these metrics after training and validation on a test dataset.\n",
        "\n",
        "In summary, while the architecture and training methodology suggest that the FJModel has the potential to yield good results, the actual performance can only be confirmed through empirical testing and evaluation on relevant datasets. If the model demonstrates low error rates and high predictive accuracy on validation data, it can be considered effective.\n",
        "\n"
      ],
      "metadata": {
        "id": "4IFvwoFuwAnZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Model Evaluation Results for FJModel\n",
        "\n",
        "The performance of the FJModel has been assessed using several key metrics, which provide insights into its predictive accuracy for insurance claims. Below are the metrics obtained after training and validating the model:\n",
        "\n",
        "1. **Mean Squared Error (MSE)**:  \n",
        "   **Value**: 24,361,113.42  \n",
        "   **Interpretation**: The MSE represents the average squared difference between the predicted values and the actual values. A lower MSE indicates better model performance; however, in this case, the MSE is relatively high, suggesting substantial errors in the predictions.\n",
        "\n",
        "2. **Mean Absolute Error (MAE)**:  \n",
        "   **Value**: 2,391.06  \n",
        "   **Interpretation**: The MAE quantifies the average absolute errors between predicted and actual values. It provides a direct measure of prediction accuracy. The value indicates that, on average, the model's predictions are off by approximately $2,391, which may be considerable depending on the context of the insurance claims.\n",
        "\n",
        "3. **R-squared Score (R²)**:  \n",
        "   **Value**: 0.19  \n",
        "   **Interpretation**: The R-squared score indicates the proportion of variance in the dependent variable that can be explained by the independent variables in the model. An R² value of 0.19 suggests that only 19% of the variability in the insurance claims can be explained by the model. This indicates that the model does not capture a significant amount of the variance, implying that it may not be performing optimally.\n",
        "\n",
        "### Summary of Results\n",
        "- **Performance Concerns**: The high MSE and MAE values, along with a low R² score, suggest that while the FJModel integrates advanced machine learning techniques, it may need further refinement or feature engineering to improve its predictive accuracy.\n",
        "- **Next Steps**: To enhance the model's performance, consider exploring the following:\n",
        "  - Feature selection and engineering to identify and incorporate more relevant predictors.\n",
        "  - Hyperparameter tuning for the DNN, TabNet, and XGBoost components to optimize their learning processes.\n",
        "  - Additional preprocessing of the dataset, including handling missing values and scaling features.\n",
        "\n",
        "These results should be documented in the report for stakeholders to understand the current effectiveness of the FJModel and potential areas for improvement.\n"
      ],
      "metadata": {
        "id": "uMRPORbrxEPh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "_7PU2-JGxEMc"
      }
    }
  ]
}