kaggle-qa-with-gemma-kerasnlp-starter-my-cut.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "800781d8",
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "papermill": {
     "duration": 0.01213,
     "end_time": "2024-03-31T10:26:30.460924",
     "exception": false,
     "start_time": "2024-03-31T10:26:30.448794",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<center><img src=\"https://keras.io/img/logo-small.png\" alt=\"Keras logo\" width=\"100\"><br/>\n",
    "This starter notebook is provided by the Keras team.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b752bc2",
   "metadata": {
    "papermill": {
     "duration": 0.01139,
     "end_time": "2024-03-31T10:26:30.484205",
     "exception": false,
     "start_time": "2024-03-31T10:26:30.472815",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<font color=\"red\"><b>Note</b>:</font><br>\n",
    "I made some changes to the original pinned notebook, to suit to my own experiments."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f20edfed",
   "metadata": {
    "papermill": {
     "duration": 0.01146,
     "end_time": "2024-03-31T10:26:30.507285",
     "exception": false,
     "start_time": "2024-03-31T10:26:30.495825",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Google – AI Assistants for Data Tasks with Gemma with [KerasNLP](https://github.com/keras-team/keras-nlp) and [Keras](https://github.com/keras-team/keras)\n",
    "\n",
    "> The objective of this competition is to build tools to assist Kaggle developers.\n",
    "\n",
    "<div align=\"center\">\n",
    "    <img src=\"https://i.ibb.co/8xZNc32/Gemma.png\">\n",
    "</div>\n",
    "\n",
    "In this competition, we are asked to create notebooks that demonstrate how to use the Gemma LLM to accomplish one or more of the following developer-oriented tasks:\n",
    "1. **<font color=\"red\">Answer common questions about the Kaggle platform.</font>**\n",
    "2. Explain or teach basic data science concepts.\n",
    "3. Summarize Kaggle Solution write-ups.\n",
    "4. Explain or teach concepts from Kaggle Solution write-ups.\n",
    "5. Answer common questions about the Python programming language.\n",
    "\n",
    "This notebook guides you through performing `\"1. Answer common questions about the Kaggle platform\"` task for the competition. As this task requires specific knowledge of Kaggle, we need precise information about Kaggle. To do so, I have created a dataset, [\"Kaggle Docs\"](https://www.kaggle.com/datasets/awsaf49/kaggle-docs), collecting data from [kaggle.com/docs](https://www.kaggle.com/docs/). To make things easier for the model, the data is curated to have Question-Answer pair format, but if you are interested, the raw data is also available. We will use this dataset to fine-tune **Gemma LLM** to answer questions about the Kaggle platform.\n",
    "\n",
    "<u>Fun fact</u>: This notebook is backend-agnostic, supporting TensorFlow, PyTorch, and JAX. However, the best performance can be achieved from `JAX`. Utilizing KerasNLP and Keras allows us to choose our preferred backend. Explore more details on [Keras](https://keras.io/keras_3/).\n",
    "\n",
    "**Note**: For a more in-depth understanding of KerasNLP, refer to the [KerasNLP guides](https://keras.io/keras_nlp/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2beb8da1",
   "metadata": {
    "papermill": {
     "duration": 0.012111,
     "end_time": "2024-03-31T10:26:30.531024",
     "exception": false,
     "start_time": "2024-03-31T10:26:30.518913",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Install Libraries  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a07819f2",
   "metadata": {
    "_kg_hide-output": true,
    "execution": {
     "iopub.execute_input": "2024-03-31T10:26:30.555510Z",
     "iopub.status.busy": "2024-03-31T10:26:30.555190Z",
     "iopub.status.idle": "2024-03-31T10:26:59.397311Z",
     "shell.execute_reply": "2024-03-31T10:26:59.396363Z"
    },
    "papermill": {
     "duration": 28.857072,
     "end_time": "2024-03-31T10:26:59.399675",
     "exception": false,
     "start_time": "2024-03-31T10:26:30.542603",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\r\n",
      "tensorflow 2.15.0 requires keras<2.16,>=2.15.0, but you have keras 3.1.1 which is incompatible.\r\n",
      "tensorflowjs 4.16.0 requires packaging~=23.1, but you have packaging 21.3 which is incompatible.\u001b[0m\u001b[31m\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# Install Keras 3 last. See https://keras.io/getting_started/ for more details.\n",
    "!pip install -q -U keras-nlp\n",
    "!pip install -q -U keras>=3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a6b9fa8",
   "metadata": {
    "papermill": {
     "duration": 0.011626,
     "end_time": "2024-03-31T10:26:59.423425",
     "exception": false,
     "start_time": "2024-03-31T10:26:59.411799",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Import Libraries "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b1ba1d7b",
   "metadata": {
    "_kg_hide-output": true,
    "execution": {
     "iopub.execute_input": "2024-03-31T10:26:59.448735Z",
     "iopub.status.busy": "2024-03-31T10:26:59.448368Z",
     "iopub.status.idle": "2024-03-31T10:27:13.372743Z",
     "shell.execute_reply": "2024-03-31T10:27:13.371947Z"
    },
    "papermill": {
     "duration": 13.940074,
     "end_time": "2024-03-31T10:27:13.375196",
     "exception": false,
     "start_time": "2024-03-31T10:26:59.435122",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-03-31 10:27:03.757997: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2024-03-31 10:27:03.758098: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2024-03-31 10:27:03.882684: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "os.environ[\"KERAS_BACKEND\"] = \"jax\" # you can also use tensorflow or torch\n",
    "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"1.00\" # avoid memory fragmentation on JAX backend.\n",
    "\n",
    "import keras\n",
    "import keras_nlp\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from tqdm.notebook import tqdm\n",
    "tqdm.pandas() # progress bar for pandas\n",
    "\n",
    "import plotly.graph_objs as go\n",
    "import plotly.express as px\n",
    "from IPython.display import display, Markdown"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f88b33b1",
   "metadata": {
    "papermill": {
     "duration": 0.011922,
     "end_time": "2024-03-31T10:27:13.399375",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.387453",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1bc8f7dd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:27:13.424530Z",
     "iopub.status.busy": "2024-03-31T10:27:13.423969Z",
     "iopub.status.idle": "2024-03-31T10:27:13.428801Z",
     "shell.execute_reply": "2024-03-31T10:27:13.428021Z"
    },
    "papermill": {
     "duration": 0.0195,
     "end_time": "2024-03-31T10:27:13.430665",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.411165",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class CFG:\n",
    "    seed = 42\n",
    "    dataset_path = \"/kaggle/input/kaggle-docs/questions_answers\"\n",
    "    preset = \"gemma_2b_en\" # name of pretrained Gemma\n",
    "    sequence_length = 512 # max size of input sequence for training\n",
    "    batch_size = 1 # size of the input batch in training, x 2 as two GPUs\n",
    "    epochs = 15 # number of epochs to train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17e53ddb",
   "metadata": {
    "papermill": {
     "duration": 0.011594,
     "end_time": "2024-03-31T10:27:13.454154",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.442560",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Reproducibility \n",
    "Sets value for random seed to produce similar result in each run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5d4422b3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:27:13.479232Z",
     "iopub.status.busy": "2024-03-31T10:27:13.478902Z",
     "iopub.status.idle": "2024-03-31T10:27:13.483387Z",
     "shell.execute_reply": "2024-03-31T10:27:13.482560Z"
    },
    "papermill": {
     "duration": 0.019224,
     "end_time": "2024-03-31T10:27:13.485285",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.466061",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "keras.utils.set_random_seed(CFG.seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a59c8e5d",
   "metadata": {
    "papermill": {
     "duration": 0.011502,
     "end_time": "2024-03-31T10:27:13.508518",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.497016",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Data\n",
    "\n",
    "The newly created **Kaggle Docs** dataset contains only approximately $60$ question-answer pairs curated from raw data from the `kaggle.com/docs` website. However, one can create many more samples from this provided data through simple augmentation or prompt engineering. For more flexibility, readers are welcome to explore the **raw** data stored in the dataset. In this notebook, we will focus on keeping it simple.\n",
    "\n",
    "**Data Format:**\n",
    "\n",
    "- The question-answer pair data is stored in `./kaggle-docs/questions_answers/data.csv` file.\n",
    "- This file includes:\n",
    "    - `Question`: A question about the Kaggle platform\n",
    "    - `Answer`: Answer to the question in markdown format\n",
    "    - `Category`: The category into which the question falls, one of the nine mentioned on the `kaggle.com/docs` website.\n",
    "    \n",
    "> You can access the **raw** data from `./kaggle-docs/raw/`, where there are `.txt` files for each of the **nine** categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c161a469",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:27:13.533541Z",
     "iopub.status.busy": "2024-03-31T10:27:13.532996Z",
     "iopub.status.idle": "2024-03-31T10:27:13.561038Z",
     "shell.execute_reply": "2024-03-31T10:27:13.560139Z"
    },
    "papermill": {
     "duration": 0.042861,
     "end_time": "2024-03-31T10:27:13.563083",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.520222",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>Answer</th>\n",
       "      <th>Category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the different types of competitions a...</td>\n",
       "      <td># Types of Competitions\\n\\nKaggle Competitions...</td>\n",
       "      <td>competition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What are the different competition formats on ...</td>\n",
       "      <td>There are handful of different formats competi...</td>\n",
       "      <td>competition</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            Question  \\\n",
       "0  What are the different types of competitions a...   \n",
       "1  What are the different competition formats on ...   \n",
       "\n",
       "                                              Answer     Category  \n",
       "0  # Types of Competitions\\n\\nKaggle Competitions...  competition  \n",
       "1  There are handful of different formats competi...  competition  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(f\"{CFG.dataset_path}/data.csv\")\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "028fca15",
   "metadata": {
    "papermill": {
     "duration": 0.01188,
     "end_time": "2024-03-31T10:27:13.588464",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.576584",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We'll use the following simple template to create prompts from question-answer pairs and category to feed text into the model:\n",
    "\n",
    "```\n",
    "Category: ...\n",
    "\n",
    "Question: ...\n",
    "\n",
    "Answer: ...\n",
    "```\n",
    "\n",
    "This template helps the model understand what you're asking and how to respond accurately. You can explore more advanced prompt templates for better results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d8e92fdf",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:27:13.613437Z",
     "iopub.status.busy": "2024-03-31T10:27:13.613126Z",
     "iopub.status.idle": "2024-03-31T10:27:13.617033Z",
     "shell.execute_reply": "2024-03-31T10:27:13.616139Z"
    },
    "papermill": {
     "duration": 0.018658,
     "end_time": "2024-03-31T10:27:13.618966",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.600308",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "template = \"\\n\\nCategory:\\nkaggle-{Category}\\n\\nQuestion:\\n{Question}\\n\\nAnswer:\\n{Answer}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "89389b81",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:27:13.644249Z",
     "iopub.status.busy": "2024-03-31T10:27:13.643567Z",
     "iopub.status.idle": "2024-03-31T10:27:13.669709Z",
     "shell.execute_reply": "2024-03-31T10:27:13.668924Z"
    },
    "papermill": {
     "duration": 0.042613,
     "end_time": "2024-03-31T10:27:13.673505",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.630892",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "38abc7f186204b7bb5fa01fb9e5858bc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/60 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df[\"prompt\"] = df.progress_apply(lambda row: template.format(Category=row.Category,\n",
    "                                                             Question=row.Question,\n",
    "                                                             Answer=row.Answer), axis=1)\n",
    "data = df.prompt.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e570fc2d",
   "metadata": {
    "papermill": {
     "duration": 0.012034,
     "end_time": "2024-03-31T10:27:13.697684",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.685650",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's examine a sample prompt. As the answers in our dataset are curated with **markdown** format, we will render the sample using `Markdown()` to properly visualize the formatting."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "161bb1cd",
   "metadata": {
    "papermill": {
     "duration": 0.011938,
     "end_time": "2024-03-31T10:27:13.721996",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.710058",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8fbf4167",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2024-03-31T10:27:13.747473Z",
     "iopub.status.busy": "2024-03-31T10:27:13.747115Z",
     "iopub.status.idle": "2024-03-31T10:27:13.752036Z",
     "shell.execute_reply": "2024-03-31T10:27:13.751187Z"
    },
    "papermill": {
     "duration": 0.019953,
     "end_time": "2024-03-31T10:27:13.753977",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.734024",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def colorize_text(text):\n",
    "    for word, color in zip([\"Category\", \"Question\", \"Answer\"], [\"blue\", \"red\", \"green\"]):\n",
    "        text = text.replace(f\"\\n\\n{word}:\", f\"\\n\\n**<font color='{color}'>{word}:</font>**\")\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "37db0275",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2024-03-31T10:27:13.779325Z",
     "iopub.status.busy": "2024-03-31T10:27:13.779038Z",
     "iopub.status.idle": "2024-03-31T10:27:13.785078Z",
     "shell.execute_reply": "2024-03-31T10:27:13.784193Z"
    },
    "papermill": {
     "duration": 0.020936,
     "end_time": "2024-03-31T10:27:13.786984",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.766048",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-competition-setup\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How do Kaggle competitions work?\n",
       "\n",
       "**<font color='green'>Answer:</font>**\n",
       "## Overview\n",
       "\n",
       "Every competition has two things:\n",
       "\n",
       "a) a clearly defined problem that participants need to solve using a machine learning model\n",
       "b) a dataset that’s used both for training and evaluating the effectiveness of these models.\n",
       "\n",
       "For example, in the [Store Sales – Time Series Forecasting](https://www.kaggle.com/competitions/store-sales-time-series-forecasting) competition, participants must accurately predict how many of each grocery item will sell using a dataset of past product and sales information from a grocery retailer.\n",
       "\n",
       "Once the competition starts, participants can submit their predictions. Kaggle will score them for accuracy, and the team will be placed on a ranked leaderboard. The team at the top of the leaderboard at the deadline wins!\n",
       "\n",
       "## Datasets, Submissions & Leaderboards\n",
       "\n",
       "Every competition’s dataset is split into two smaller datasets.\n",
       "\n",
       "- One of these smaller datasets will be given to participants to train their models, typically named `train.csv`.\n",
       "- The other dataset will be mostly hidden from participants and used by Kaggle for testing and scoring, named `test.csv` and `solution.csv` (`test.csv` is the same as `solution.csv` except that `test.csv` contains the feature values and `solution.csv` contains the ground truth variable(s) – participants will never, ever see `solution.csv`).\n",
       "\n",
       "When a participant feels ready to make a submission to the competition, they will use `test.csv` to generate a prediction and upload a CSV file. Kaggle will automatically score the submission for accuracy using the hidden `solution.csv` file.\n",
       "\n",
       "Most competitions have a maximum number of submissions that a participant can make each day and a final deadline at which point the leaderboard will be frozen.\n",
       "\n",
       "It’s conceivable that a participant could use the mechanics of a Kaggle competition to overfit a solution - which would be great for winning a competition, but not valuable for a real-world application.\n",
       "\n",
       "To help prevent this, Kaggle has two leaderboards – the public and private leaderboard. The competition host splits the `solution.csv` dataset into two parts, using one part for the public leaderboard and another part for the private leaderboard. Participants generally will now know which samples are public vs private. The private leaderboard is kept a secret until after the competition deadline and is used as the official leaderboard for determining the final ranking."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Take a random sample\n",
    "sample = data[45]\n",
    "\n",
    "# Give colors to Question, Answer and Category\n",
    "sample = colorize_text(sample)\n",
    "\n",
    "# Show sample in markdown\n",
    "display(Markdown(sample))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdf9fbba",
   "metadata": {
    "papermill": {
     "duration": 0.012144,
     "end_time": "2024-03-31T10:27:13.811646",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.799502",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Exploratory Data Analysis\n",
    "\n",
    "Let's do a simple EDA to determine how many question-answer pairs we have per category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7551e478",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2024-03-31T10:27:13.839921Z",
     "iopub.status.busy": "2024-03-31T10:27:13.839433Z",
     "iopub.status.idle": "2024-03-31T10:27:14.090046Z",
     "shell.execute_reply": "2024-03-31T10:27:14.089132Z"
    },
    "papermill": {
     "duration": 0.266452,
     "end_time": "2024-03-31T10:27:14.092021",
     "exception": false,
     "start_time": "2024-03-31T10:27:13.825569",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "        <script type=\"text/javascript\">\n",
       "        window.PlotlyConfig = {MathJaxConfig: 'local'};\n",
       "        if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}\n",
       "        if (typeof require !== 'undefined') {\n",
       "        require.undef(\"plotly\");\n",
       "        requirejs.config({\n",
       "            paths: {\n",
       "                'plotly': ['https://cdn.plot.ly/plotly-2.27.0.min']\n",
       "            }\n",
       "        });\n",
       "        require(['plotly'], function(Plotly) {\n",
       "            window._Plotly = Plotly;\n",
       "        });\n",
       "        }\n",
       "        </script>\n",
       "        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>                            <div id=\"2c3f04a6-30eb-4080-931e-3516fbe91fbc\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"2c3f04a6-30eb-4080-931e-3516fbe91fbc\")) {                    Plotly.newPlot(                        \"2c3f04a6-30eb-4080-931e-3516fbe91fbc\",                        [{\"x\":[\"api\",\"competition\",\"competition-setup\",\"dataset\",\"gpu\",\"model\",\"noteboook\",\"organization\",\"tpu\"],\"y\":[4,8,10,6,1,6,11,5,9],\"type\":\"bar\",\"text\":[4.0,8.0,10.0,6.0,1.0,6.0,11.0,5.0,9.0],\"textposition\":\"outside\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"title\":{\"text\":\"Category Distribution\"},\"xaxis\":{\"title\":{\"text\":\"Category\"}},\"yaxis\":{\"title\":{\"text\":\"Count\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('2c3f04a6-30eb-4080-931e-3516fbe91fbc');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get unique labels and their frequency\n",
    "unique_labels, label_counts = np.unique(df.Category.tolist(), return_counts=True)\n",
    "\n",
    "# Plotting\n",
    "fig = go.Figure(data=go.Bar(x=unique_labels, y=label_counts))\n",
    "fig.update_layout(\n",
    "    title=\"Category Distribution\",\n",
    "    xaxis_title=\"Category\",\n",
    "    yaxis_title=\"Count\",\n",
    ")\n",
    "\n",
    "fig.update_traces(text=label_counts, textposition=\"outside\")\n",
    "fig.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a6ef0d2",
   "metadata": {
    "papermill": {
     "duration": 0.012706,
     "end_time": "2024-03-31T10:27:14.158878",
     "exception": false,
     "start_time": "2024-03-31T10:27:14.146172",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Modeling\n",
    "\n",
    "<div align=\"center\"><img src=\"https://i.ibb.co/Bqg9w3g/Gemma-Logo-no-background.png\" width=\"300\"></div>\n",
    "\n",
    "**Gemma** is a suite of advanced open models developed by **Google DeepMind** and other **Google teams**, derived from the same research and technology behind the **Gemini** models. They can be integrated into applications and run on various platforms including mobile devices and hosted services. Developers can customize Gemma models using tuning techniques to enhance their performance for specific tasks, offering more targeted and efficient generative AI solutions beyond text generation.\n",
    "\n",
    "Gemma models are available in several sizes so you can build generative AI solutions based on your available computing resources, the capabilities you need, and where you want to run them.\n",
    "\n",
    "| Parameters size | Tuned versions    | Intended platforms                 | Preset                 |\n",
    "|-----------------|-------------------|------------------------------------|------------------------|\n",
    "| 2B              | Pretrained        | Mobile devices and laptops         | `gemma_2b_en`          |\n",
    "| 2B              | Instruction tuned | Mobile devices and laptops         | `gemma_instruct_2b_en` |\n",
    "| 7B              | Pretrained        | Desktop computers and small servers| `gemma_7b_en`          |\n",
    "| 7B              | Instruction tuned | Desktop computers and small servers| `gemma_instruct_7b_en` |\n",
    "\n",
    "In this notebook, we will use the `Gemma 2B` from KerasNLP's pretrained models to answer questions about the Kaggle platform. To explore other models, simply modify the `preset` in the `CFG` (config). A list of other available pretrained models can be found on the [KerasNLP website](https://keras.io/api/keras_nlp/models/).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3eba00f",
   "metadata": {
    "papermill": {
     "duration": 0.012593,
     "end_time": "2024-03-31T10:27:14.184262",
     "exception": false,
     "start_time": "2024-03-31T10:27:14.171669",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Gemma Causal LM\n",
    "\n",
    "The code below will build an end-to-end Gemma model for causal language modeling (hence the name `GemmaCausalLM`). A causal language model (LM) predicts the next token based on previous tokens. This task setup can be used to train the model unsupervised on plain text input or to autoregressively generate plain text similar to the data used for training. This task can be used for pre-training or fine-tuning a Gemma model simply by calling `fit()`.\n",
    "\n",
    "This model has a `generate()` method, which generates text based on a prompt. The generation strategy used is controlled by an additional sampler argument on `compile()`. You can recompile the model with different `keras_nlp.samplers` objects to control the generation. By default, `\"greedy\"` sampling will be used.\n",
    "\n",
    "> The `from_preset` method instantiates the model from a preset architecture and weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b21ab2ba",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:27:14.211592Z",
     "iopub.status.busy": "2024-03-31T10:27:14.211243Z",
     "iopub.status.idle": "2024-03-31T10:28:05.485022Z",
     "shell.execute_reply": "2024-03-31T10:28:05.484155Z"
    },
    "papermill": {
     "duration": 51.290005,
     "end_time": "2024-03-31T10:28:05.486976",
     "exception": false,
     "start_time": "2024-03-31T10:27:14.196971",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Attaching 'config.json' from model 'keras/gemma/keras/gemma_2b_en/2' to your Kaggle notebook...\n",
      "Attaching 'config.json' from model 'keras/gemma/keras/gemma_2b_en/2' to your Kaggle notebook...\n",
      "Attaching 'model.weights.h5' from model 'keras/gemma/keras/gemma_2b_en/2' to your Kaggle notebook...\n",
      "Attaching 'tokenizer.json' from model 'keras/gemma/keras/gemma_2b_en/2' to your Kaggle notebook...\n",
      "Attaching 'assets/tokenizer/vocabulary.spm' from model 'keras/gemma/keras/gemma_2b_en/2' to your Kaggle notebook...\n",
      "normalizer.cc(51) LOG(INFO) precompiled_charsmap is empty. use identity normalization.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"gemma_causal_lm_preprocessor\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Tokenizer (type)                                   </span>┃<span style=\"font-weight: bold\">                                             Vocab # </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaTokenizer</span>)                   │                                             <span style=\"color: #00af00; text-decoration-color: #00af00\">256,000</span> │\n",
       "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mTokenizer (type)                                  \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                            Vocab #\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m)                   │                                             \u001b[38;5;34m256,000\u001b[0m │\n",
       "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"gemma_causal_lm\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ gemma_backbone                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2048</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,506,172,416</span> │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaBackbone</span>)               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256000</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">524,288,000</span> │ gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
       "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ gemma_backbone                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2048\u001b[0m)        │   \u001b[38;5;34m2,506,172,416\u001b[0m │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│ (\u001b[38;5;33mGemmaBackbone\u001b[0m)               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m)      │     \u001b[38;5;34m524,288,000\u001b[0m │ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n",
       "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
       "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,506,172,416</span> (9.34 GB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,506,172,416\u001b[0m (9.34 GB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,506,172,416</span> (9.34 GB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,506,172,416\u001b[0m (9.34 GB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(\"gemma_2b_en\")\n",
    "gemma_lm.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3452329a",
   "metadata": {
    "papermill": {
     "duration": 0.014359,
     "end_time": "2024-03-31T10:28:05.516155",
     "exception": false,
     "start_time": "2024-03-31T10:28:05.501796",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Gemma LM Preprocessor\n",
    "\n",
    "An important part of the Gemma model is the **Preprocessor** layer, which under the hood uses **Tokenizer**.\n",
    "\n",
    "**What it does:** The preprocessor takes input strings and transforms them into a dictionary (`token_ids`, `padding_mask`) containing preprocessed tensors. This process starts with tokenization, where input strings are converted into sequences of token IDs.\n",
    "\n",
    "**Why it's important:** Initially, raw text data is complex and challenging for modeling due to its high dimensionality. By converting text into a compact set of tokens, such as transforming `\"The quick brown fox\"` into `[\"the\", \"qu\", \"##ick\", \"br\", \"##own\", \"fox\"]`, we simplify the data. Many models rely on special tokens and additional tensors to understand input. These tokens help divide input and identify padding, among other tasks. Making all sequences the same length through padding boosts computational efficiency, making subsequent steps smoother.\n",
    "\n",
    "Explore the following pages to access the available preprocessing and tokenizer layers in **KerasNLP**:\n",
    "- [Preprocessing](https://keras.io/api/keras_nlp/preprocessing_layers/)\n",
    "- [Tokenizers](https://keras.io/api/keras_nlp/tokenizers/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0ecdf275",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:28:05.546504Z",
     "iopub.status.busy": "2024-03-31T10:28:05.546173Z",
     "iopub.status.idle": "2024-03-31T10:28:05.886672Z",
     "shell.execute_reply": "2024-03-31T10:28:05.885861Z"
    },
    "papermill": {
     "duration": 0.358454,
     "end_time": "2024-03-31T10:28:05.889056",
     "exception": false,
     "start_time": "2024-03-31T10:28:05.530602",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "x, y, sample_weight = gemma_lm.preprocessor(data[0:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c17b39ca",
   "metadata": {
    "papermill": {
     "duration": 0.014212,
     "end_time": "2024-03-31T10:28:05.918068",
     "exception": false,
     "start_time": "2024-03-31T10:28:05.903856",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "This preprocessing layer will take in batches of strings, and return outputs in a `(x, y, sample_weight)` format, where the `y` label is the next token id in the `x` sequence.\n",
    "\n",
    "From the code below, we can see that, after the preprocessor, the data shape is `(num_samples, sequence_length)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "84b7b903",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:28:05.948293Z",
     "iopub.status.busy": "2024-03-31T10:28:05.947975Z",
     "iopub.status.idle": "2024-03-31T10:28:05.952977Z",
     "shell.execute_reply": "2024-03-31T10:28:05.952135Z"
    },
    "papermill": {
     "duration": 0.022201,
     "end_time": "2024-03-31T10:28:05.954933",
     "exception": false,
     "start_time": "2024-03-31T10:28:05.932732",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "token_ids : (2, 8192)\n",
      "padding_mask : (2, 8192)\n"
     ]
    }
   ],
   "source": [
    "# Display the shape of each processed output\n",
    "for k, v in x.items():\n",
    "    print(k, \":\", v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "894b3d1d",
   "metadata": {
    "papermill": {
     "duration": 0.014176,
     "end_time": "2024-03-31T10:28:05.983572",
     "exception": false,
     "start_time": "2024-03-31T10:28:05.969396",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Inference before fine tuning\n",
    "\n",
    "Let's ask the Gemma model some sample questions using our prepared prompt and see how it responds. \n",
    "\n",
    "> As this model is not tuned for instruction yet, you will notice that the model is creating more question-answer pairs instead of answering the question that was asked."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3405a59",
   "metadata": {
    "papermill": {
     "duration": 0.014156,
     "end_time": "2024-03-31T10:28:06.012134",
     "exception": false,
     "start_time": "2024-03-31T10:28:05.997978",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Sample 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "700b1735",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2024-03-31T10:28:06.042430Z",
     "iopub.status.busy": "2024-03-31T10:28:06.041834Z",
     "iopub.status.idle": "2024-03-31T10:28:22.910504Z",
     "shell.execute_reply": "2024-03-31T10:28:22.909538Z"
    },
    "papermill": {
     "duration": 16.886138,
     "end_time": "2024-03-31T10:28:22.912649",
     "exception": false,
     "start_time": "2024-03-31T10:28:06.026511",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-competition\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How to join a competition?\n",
       "\n",
       "**<font color='green'>Answer:</font>**\n",
       "1. Go to the competition page.\n",
       "2. Click on the \"Join\" button.\n",
       "3. Enter your email address and click on the \"Join\" button.\n",
       "4. You will receive an email with a link to confirm your email address.\n",
       "5. Click on the link in the email to confirm your email address.\n",
       "6. You will now be able to log in to the competition.\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-competition\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How to submit a solution?\n",
       "\n",
       "**<font color='green'>Answer:</font>**\n",
       "1. Go to the competition page.\n",
       "2. Click on the \"Submit\" button.\n",
       "3. Enter your solution in the text box and click on the \"Submit\" button.\n",
       "4. You will receive a confirmation email with the status of your submission.\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-competition\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How to view the leaderboard?\n",
       "\n",
       "**<font color='green'>Answer:</font>**\n",
       "1. Go to the competition page.\n",
       "2. Click on the \"Leaderboard\" button.\n",
       "3. You will see the leaderboard with the top 100 participants.\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-competition\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How to view the"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Take one sample\n",
    "row = df.iloc[2]\n",
    "\n",
    "# Generate Prompt using template\n",
    "prompt = template.format(\n",
    "    Category=row.Category,\n",
    "    Question=row.Question,\n",
    "    Answer=\"\"\n",
    ")\n",
    "\n",
    "# Infer\n",
    "output = gemma_lm.generate(prompt, max_length=256)\n",
    "\n",
    "# Colorize\n",
    "output = colorize_text(output)\n",
    "\n",
    "# Display in markdown\n",
    "display(Markdown(output))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44d7bcb6",
   "metadata": {
    "papermill": {
     "duration": 0.014426,
     "end_time": "2024-03-31T10:28:22.942047",
     "exception": false,
     "start_time": "2024-03-31T10:28:22.927621",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Sample 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "633bfc8a",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2024-03-31T10:28:22.973681Z",
     "iopub.status.busy": "2024-03-31T10:28:22.973355Z",
     "iopub.status.idle": "2024-03-31T10:28:28.614651Z",
     "shell.execute_reply": "2024-03-31T10:28:28.613651Z"
    },
    "papermill": {
     "duration": 5.659775,
     "end_time": "2024-03-31T10:28:28.616940",
     "exception": false,
     "start_time": "2024-03-31T10:28:22.957165",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-competition-setup\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How do Kaggle competitions work?\n",
       "\n",
       "**<font color='green'>Answer:</font>**\n",
       "Kaggle competitions are a way for Kaggle users to compete against each other and win prizes.\n",
       "\n",
       "To participate in a competition, you must first create an account on Kaggle. Once you have an account, you can start competing by creating a project.\n",
       "\n",
       "A project is a collection of notebooks that you can use to solve a problem. You can create a project from scratch or use one of the many templates that are available.\n",
       "\n",
       "Once you have created a project, you can start working on it. You can use the notebooks in your project to solve the problem, or you can create new notebooks to solve the problem.\n",
       "\n",
       "When you are finished working on your project, you can submit it to the competition. The competition will then review your project and decide if you have solved the problem correctly.\n",
       "\n",
       "If you are successful in solving the problem, you will be awarded a prize.\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-competition-setup\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How do I create a project?\n",
       "\n",
       "**<font color='green'>Answer:</font>**\n",
       "To create a project, you must first create an account on Kaggle. Once you have an account, you can start creating projects by clicking"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Take one sample\n",
    "row = df.iloc[45]\n",
    "\n",
    "# Generate Prompt using template\n",
    "prompt = template.format(\n",
    "    Category=row.Category,\n",
    "    Question=row.Question,\n",
    "    Answer=\"\"\n",
    ")\n",
    "\n",
    "# Infer\n",
    "output = gemma_lm.generate(prompt, max_length=256)\n",
    "\n",
    "# Colorize\n",
    "output = colorize_text(output)\n",
    "\n",
    "# Display in markdown\n",
    "display(Markdown(output))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee9ea74a",
   "metadata": {
    "papermill": {
     "duration": 0.014663,
     "end_time": "2024-03-31T10:28:28.646995",
     "exception": false,
     "start_time": "2024-03-31T10:28:28.632332",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Fine-tuning with LoRA\n",
    "\n",
    "To get better responses from the model, we will fine-tune the model with Low Rank Adaptation (LoRA) on the **Kaggle Docs** dataset.\n",
    "\n",
    "**What exactly is LoRA?**\n",
    "\n",
    "LoRA is a method used to fine-tune large language models (LLMs) in an efficient way. It involves freezing the weights of the LLM and injecting trainable rank-decomposition matrices.\n",
    "\n",
    "Imagine in an LLM, we have a pre-trained dense layer, represented by a $d \\times d$ weight matrix, denoted as $W_0$. We then initialize two additional dense layers, labeled as $A$ and $B$, with shapes $d \\times r$ and $r \\times d$, respectively. Here, $r$ denotes the rank, which is typically **much smaller than** $d$. Prior to LoRA, the model's output was computed using the equation $output = W_0 \\cdot x + b_0$, where $x$ represents the input and $b_0$ denotes the bias term associated with the original dense layer, which remains frozen. After applying LoRA, the equation becomes $output = (W_0 \\cdot x + b_0) + (B \\cdot A \\cdot x)$, where $A$ and $B$ denote the trainable rank-decomposition matrices that have been introduced.\n",
    "\n",
    "<center><img src=\"https://i.ibb.co/DWsbhLg/LoRA.png\" width=\"300\"><br/>\n",
    "Credit: <a href=\"https://arxiv.org/abs/2106.09685\">LoRA: Low-Rank Adaptation of Large Language Models</a> Paper</center>\n",
    "\n",
    "\n",
    "In the paper, $A$ is initialized with $\\mathcal{N} (0, \\sigma^2)$ and $B$ with $0$, where $\\mathcal{N}$ denotes the normal distribution, and $\\sigma^2$ is the variance.\n",
    "\n",
    "**Why does LoRA save memory?**\n",
    "\n",
    "Even though we're adding more layers to the model with LoRA, it actually helps save memory. This is because the smaller layers (A and B) have fewer parameters to learn compared to the big model and fewer trainable parameters mean fewer optimizer variables to store. So, even though the overall model might seem bigger, it's actually more efficient in terms of memory usage. \n",
    "\n",
    "> This notebook uses a LoRA rank of `4`. A higher rank means more detailed changes are possible, but also means more trainable parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "12f29e31",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:28:28.677395Z",
     "iopub.status.busy": "2024-03-31T10:28:28.677072Z",
     "iopub.status.idle": "2024-03-31T10:28:29.145993Z",
     "shell.execute_reply": "2024-03-31T10:28:29.145036Z"
    },
    "papermill": {
     "duration": 0.486565,
     "end_time": "2024-03-31T10:28:29.148126",
     "exception": false,
     "start_time": "2024-03-31T10:28:28.661561",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"gemma_causal_lm_preprocessor\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Tokenizer (type)                                   </span>┃<span style=\"font-weight: bold\">                                             Vocab # </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaTokenizer</span>)                   │                                             <span style=\"color: #00af00; text-decoration-color: #00af00\">256,000</span> │\n",
       "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mTokenizer (type)                                  \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                            Vocab #\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m)                   │                                             \u001b[38;5;34m256,000\u001b[0m │\n",
       "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"gemma_causal_lm\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ gemma_backbone                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2048</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,507,536,384</span> │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaBackbone</span>)               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256000</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">524,288,000</span> │ gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
       "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ gemma_backbone                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2048\u001b[0m)        │   \u001b[38;5;34m2,507,536,384\u001b[0m │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│ (\u001b[38;5;33mGemmaBackbone\u001b[0m)               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
       "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
       "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m)      │     \u001b[38;5;34m524,288,000\u001b[0m │ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n",
       "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
       "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,507,536,384</span> (9.34 GB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,507,536,384\u001b[0m (9.34 GB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">1,363,968</span> (5.20 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m1,363,968\u001b[0m (5.20 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,506,172,416</span> (9.34 GB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m2,506,172,416\u001b[0m (9.34 GB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Enable LoRA for the model and set the LoRA rank to 4.\n",
    "gemma_lm.backbone.enable_lora(rank=4)\n",
    "gemma_lm.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d26d5e4a",
   "metadata": {
    "papermill": {
     "duration": 0.01621,
     "end_time": "2024-03-31T10:28:29.181618",
     "exception": false,
     "start_time": "2024-03-31T10:28:29.165408",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Notice** that, the number of trainable parameters is reduced from ~$2.5$ billions to ~$1.3$ millions after enabling LoRA."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91a56fda",
   "metadata": {
    "papermill": {
     "duration": 0.017107,
     "end_time": "2024-03-31T10:28:29.214948",
     "exception": false,
     "start_time": "2024-03-31T10:28:29.197841",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c7d2d5b4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:28:29.248083Z",
     "iopub.status.busy": "2024-03-31T10:28:29.247763Z",
     "iopub.status.idle": "2024-03-31T10:41:07.173905Z",
     "shell.execute_reply": "2024-03-31T10:41:07.172794Z"
    },
    "papermill": {
     "duration": 757.945309,
     "end_time": "2024-03-31T10:41:07.176139",
     "exception": false,
     "start_time": "2024-03-31T10:28:29.230830",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m65s\u001b[0m 734ms/step - loss: 1.7209 - sparse_categorical_accuracy: 0.5241\n",
      "Epoch 2/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m44s\u001b[0m 727ms/step - loss: 1.6869 - sparse_categorical_accuracy: 0.5313\n",
      "Epoch 3/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m44s\u001b[0m 728ms/step - loss: 1.6175 - sparse_categorical_accuracy: 0.5417\n",
      "Epoch 4/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m44s\u001b[0m 727ms/step - loss: 1.5770 - sparse_categorical_accuracy: 0.5509\n",
      "Epoch 5/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m44s\u001b[0m 728ms/step - loss: 1.5537 - sparse_categorical_accuracy: 0.5552\n",
      "Epoch 6/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m44s\u001b[0m 727ms/step - loss: 1.5304 - sparse_categorical_accuracy: 0.5568\n",
      "Epoch 7/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m83s\u001b[0m 1s/step - loss: 1.5028 - sparse_categorical_accuracy: 0.5630\n",
      "Epoch 8/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m44s\u001b[0m 727ms/step - loss: 1.4733 - sparse_categorical_accuracy: 0.5682\n",
      "Epoch 9/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m44s\u001b[0m 728ms/step - loss: 1.4444 - sparse_categorical_accuracy: 0.5747\n",
      "Epoch 10/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m83s\u001b[0m 1s/step - loss: 1.4025 - sparse_categorical_accuracy: 0.5873\n",
      "Epoch 11/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m44s\u001b[0m 727ms/step - loss: 1.3607 - sparse_categorical_accuracy: 0.5960\n",
      "Epoch 12/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m44s\u001b[0m 728ms/step - loss: 1.3163 - sparse_categorical_accuracy: 0.6080\n",
      "Epoch 13/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m44s\u001b[0m 728ms/step - loss: 1.2683 - sparse_categorical_accuracy: 0.6199\n",
      "Epoch 14/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m44s\u001b[0m 728ms/step - loss: 1.2149 - sparse_categorical_accuracy: 0.6325\n",
      "Epoch 15/15\n",
      "\u001b[1m60/60\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m44s\u001b[0m 728ms/step - loss: 1.1584 - sparse_categorical_accuracy: 0.6451\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.history.History at 0x7af3b02ffd00>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Limit the input sequence length to 512 (to control memory usage).\n",
    "gemma_lm.preprocessor.sequence_length = CFG.sequence_length \n",
    "\n",
    "# Compile the model with loss, optimizer, and metric\n",
    "gemma_lm.compile(\n",
    "    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "    optimizer=keras.optimizers.Adam(learning_rate=8e-5),\n",
    "    weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],\n",
    ")\n",
    "\n",
    "# Train model\n",
    "gemma_lm.fit(data, epochs=CFG.epochs, batch_size=CFG.batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a7deaad",
   "metadata": {
    "papermill": {
     "duration": 0.088215,
     "end_time": "2024-03-31T10:41:07.354918",
     "exception": false,
     "start_time": "2024-03-31T10:41:07.266703",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Inference after fine-tuning\n",
    "\n",
    "Let's see how our fine-tuned model responds to the same questions we asked before fine-tuning the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d8c6605",
   "metadata": {
    "papermill": {
     "duration": 0.087266,
     "end_time": "2024-03-31T10:41:07.529885",
     "exception": false,
     "start_time": "2024-03-31T10:41:07.442619",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Sample 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "454a1cc6",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2024-03-31T10:41:07.707745Z",
     "iopub.status.busy": "2024-03-31T10:41:07.707134Z",
     "iopub.status.idle": "2024-03-31T10:41:24.157920Z",
     "shell.execute_reply": "2024-03-31T10:41:24.156963Z"
    },
    "papermill": {
     "duration": 16.54171,
     "end_time": "2024-03-31T10:41:24.159914",
     "exception": false,
     "start_time": "2024-03-31T10:41:07.618204",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-competition\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How to join a competition?\n",
       "\n",
       "**<font color='green'>Answer:</font>**\n",
       "You need to sign in to your [Kaggle account](https://www.kaggle.com/signup) to join a competition.\n",
       "\n",
       "You’ll then be redirected to the competition page, where you can either:\n",
       "\n",
       "- Click “Join” to become a participant in that competition. This will allow you to download the data and start working on your model.\n",
       "\n",
       "- Click \"Download and upload the data and then click \"Join Leaderboard\" to view the rules of participating in the competition.\n",
       "\n",
       "- Click “Download Data” to download a copy of the competition’s data. You’ll need to do this before you can start working on your model.\n",
       "\n",
       "If you have any questions about this process, feel free to reach out to our support team [here](https://www.kaggle.com/support/contact/Contact)."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Take one sample\n",
    "row = df.iloc[2]\n",
    "\n",
    "# Generate Prompt using template\n",
    "prompt = template.format(\n",
    "    Category=row.Category,\n",
    "    Question=row.Question,\n",
    "    Answer=\"\"\n",
    ")\n",
    "\n",
    "# Infer\n",
    "output = gemma_lm.generate(prompt, max_length=256)\n",
    "\n",
    "# Colorize\n",
    "output = colorize_text(output)\n",
    "\n",
    "# Display in markdown\n",
    "display(Markdown(output))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "337c153f",
   "metadata": {
    "papermill": {
     "duration": 0.089195,
     "end_time": "2024-03-31T10:41:24.342081",
     "exception": false,
     "start_time": "2024-03-31T10:41:24.252886",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Sample 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "66d59d15",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2024-03-31T10:41:24.522495Z",
     "iopub.status.busy": "2024-03-31T10:41:24.522123Z",
     "iopub.status.idle": "2024-03-31T10:41:30.961952Z",
     "shell.execute_reply": "2024-03-31T10:41:30.961085Z"
    },
    "papermill": {
     "duration": 6.532304,
     "end_time": "2024-03-31T10:41:30.963953",
     "exception": false,
     "start_time": "2024-03-31T10:41:24.431649",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-competition-setup\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How do Kaggle competitions work?\n",
       "\n",
       "**<font color='green'>Answer:</font>**\n",
       "When you click “Open dataset” on any public competition, you’ll be taken to the “Overview” tab. There you’ll find details about the problem and any data that is needed to solve it.\n",
       "\n",
       "The \"Overview\" is broken into different sections which we’ll cover in more detail. The different parts of this section are:\n",
       "\n",
       "- **Problem**: The problem statement.\n",
       "- **Competition rules**: The competition rules that govern your participation.\n",
       "- **Dataset**: A brief description of the dataset, including the number of files and the number of samples.\n",
       "- **Data sources and licenses**: Details on where the data came from and what the licensing terms were.\n",
       "\n",
       "The \"Files\" section of the overview is also very important. It contains the details about how the data is formatted and what the data looks like.\n",
       "\n",
       "- **Training dataset**: The number of samples and the number of files in this folder will match the information that you see on the \"Overview\" tab.\n",
       "- **Evaluation dataset**: The number of samples and the number of files in this folder will also match the information that you see on the \""
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Take one sample\n",
    "row = df.iloc[45]\n",
    "\n",
    "# Generate Prompt using template\n",
    "prompt = template.format(\n",
    "    Category=row.Category,\n",
    "    Question=row.Question,\n",
    "    Answer=\"\"\n",
    ")\n",
    "\n",
    "# Infer\n",
    "output = gemma_lm.generate(prompt, max_length=256)\n",
    "\n",
    "# Colorize\n",
    "output = colorize_text(output)\n",
    "\n",
    "# Display in markdown\n",
    "display(Markdown(output))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20280dc6",
   "metadata": {
    "papermill": {
     "duration": 0.101745,
     "end_time": "2024-03-31T10:41:31.157694",
     "exception": false,
     "start_time": "2024-03-31T10:41:31.055949",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Sample 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7af20955",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:41:31.341883Z",
     "iopub.status.busy": "2024-03-31T10:41:31.341488Z",
     "iopub.status.idle": "2024-03-31T10:41:37.835574Z",
     "shell.execute_reply": "2024-03-31T10:41:37.834659Z"
    },
    "papermill": {
     "duration": 6.58824,
     "end_time": "2024-03-31T10:41:37.837671",
     "exception": false,
     "start_time": "2024-03-31T10:41:31.249431",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-model\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How to find Kaggle Models?\n",
       "\n",
       "**<font color='green'>Answer:</font>**\n",
       "There are a number of ways for you to discover, explore, and get access to the wide range of models available on Kaggle.\n",
       "\n",
       "First and foremost, we highly encourage you to check out the [Featured Models page](https://www.kaggle.com/featured). It showcases the latest models in each model category from the past seven days. This is a great place to get started if you want to see the hot new models in the field.\n",
       "\n",
       "Next, we have the [Model Spotlight](https://www.kaggle.com/modelspotlight) and the [Model Spotlight Leaderboard](https://www.kaggle.com/modelspotlightleaderboard). These are collections of the most downloaded and most upvoted models on Kaggle. The model spotlight leaderboards are refreshed every 24 hours and feature the top 25 models in each model category. The model spotlight is a great place to get a quick overview of the top models in each category. It’s also a good way to see what models are popular and trending in a particular category.\n",
       "\n",
       "Finally, you can search for models using Kaggle’s [Datasets page]("
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Take one sample\n",
    "row = df.iloc[20]\n",
    "\n",
    "# Generate Prompt using template\n",
    "prompt = template.format(\n",
    "    Category=row.Category,\n",
    "    Question=row.Question,\n",
    "    Answer=\"\"\n",
    ")\n",
    "\n",
    "# Infer\n",
    "output = gemma_lm.generate(prompt, max_length=256)\n",
    "\n",
    "# Colorize\n",
    "output = colorize_text(output)\n",
    "\n",
    "# Display in markdown\n",
    "display(Markdown(output))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f44d4db3",
   "metadata": {
    "papermill": {
     "duration": 0.091106,
     "end_time": "2024-03-31T10:41:38.020128",
     "exception": false,
     "start_time": "2024-03-31T10:41:37.929022",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Unseen Sample(s)\n",
    "\n",
    "Also just for fun, let's try out a question that model hasn't seen during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1f778e5c",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2024-03-31T10:41:38.200754Z",
     "iopub.status.busy": "2024-03-31T10:41:38.200070Z",
     "iopub.status.idle": "2024-03-31T10:41:42.464013Z",
     "shell.execute_reply": "2024-03-31T10:41:42.463062Z"
    },
    "papermill": {
     "duration": 4.357207,
     "end_time": "2024-03-31T10:41:42.466158",
     "exception": false,
     "start_time": "2024-03-31T10:41:38.108951",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-kaggle-notebook\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How to export a notebook?\n",
       "\n",
       "**<font color='green'>Answer:</font>**\n",
       "You can always save a notebook locally from the “Save” button in the top right corner.\n",
       "\n",
       "However, the local copy of the notebook is not shared with other Kaggle users. If you need a permanent record of your work, or you need your notebook to work on other platforms such as GitHub, you should use one of the following options:\n",
       "\n",
       "1) Save the notebook to your Google Drive. You can find it there at a convenient location.\n",
       "\n",
       "2) Share the notebook URL on Kaggle, which will open the notebook in a new tab and allow others to view and comment on it. This is the default way notebooks are displayed on Kaggle.\n",
       "\n",
       "3) Export the notebook as a zip file to download and host locally."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Generate Prompt using template\n",
    "prompt = template.format(\n",
    "    Category=\"kaggle-notebook\",\n",
    "    Question=\"How to export a notebook?\",\n",
    "    Answer=\"\"\n",
    ")\n",
    "\n",
    "# Infer\n",
    "output = gemma_lm.generate(prompt, max_length=256)\n",
    "\n",
    "# Colorize\n",
    "output = colorize_text(output)\n",
    "\n",
    "# Display in markdown\n",
    "display(Markdown(output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "5cfb7d8d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:41:42.648871Z",
     "iopub.status.busy": "2024-03-31T10:41:42.648547Z",
     "iopub.status.idle": "2024-03-31T10:41:48.978187Z",
     "shell.execute_reply": "2024-03-31T10:41:48.977114Z"
    },
    "papermill": {
     "duration": 6.423162,
     "end_time": "2024-03-31T10:41:48.980691",
     "exception": false,
     "start_time": "2024-03-31T10:41:42.557529",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-kaggle-notebook\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How to create a notebook using Kaggle API?\n",
       "\n",
       "**<font color='green'>Answer:</font>**\n",
       "Creating a Notebook using Kaggle API:\n",
       "\n",
       "You can create a notebook using Kaggle API. This allows you to create notebooks quickly from the command line.\n",
       "\n",
       "To use this endpoint, you need to be a project owner or a notebook author of the Notebook project you want to upload the notebook to.\n",
       "\n",
       "You can find the URL for a Notebook project by clicking the “Share” button in the top right corner of any Notebook page.\n",
       "\n",
       "You can find the URL for a specific Notebook by clicking the Notebook name from the list on the Project details page.\n",
       "\n",
       "You can find the list of all Notebook projects on the Kaggle website: https://kaggle.com/projects/all.\n",
       "\n",
       "You can use the following API endpoint to create a Notebook:\n",
       "\n",
       "```\n",
       "https://kaggle-api.s3-us-west-1.amazonaws.com/projects?access_token=<your token>\n",
       "```\n",
       "\n",
       "Replace `<your token>` with the access token for the project you want to upload the notebook to.\n",
       "\n",
       "You can find your access token from your user profile. You should also store your access token in"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Generate Prompt using template\n",
    "prompt = template.format(\n",
    "    Category=\"kaggle-notebook\",\n",
    "    Question=\"How to create a notebook using Kaggle API?\",\n",
    "    Answer=\"\"\n",
    ")\n",
    "\n",
    "# Infer\n",
    "output = gemma_lm.generate(prompt, max_length=256)\n",
    "\n",
    "# Colorize\n",
    "output = colorize_text(output)\n",
    "\n",
    "# Display in markdown\n",
    "display(Markdown(output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0058aac5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:41:49.165804Z",
     "iopub.status.busy": "2024-03-31T10:41:49.165430Z",
     "iopub.status.idle": "2024-03-31T10:41:54.681738Z",
     "shell.execute_reply": "2024-03-31T10:41:54.680726Z"
    },
    "papermill": {
     "duration": 5.609696,
     "end_time": "2024-03-31T10:41:54.683808",
     "exception": false,
     "start_time": "2024-03-31T10:41:49.074112",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-kaggle-competitions\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How you can create a in-class competition?\n",
       "\n",
       "**<font color='green'>Answer:</font>**\n",
       "There are two ways to run a competition on Kaggle.\n",
       "\n",
       "First, you can run an open competition where anyone can participate and submit solutions. This is great if you want to gather a large amount of data to train a model on, but not ideal for a private challenge where you want to control who can participate or view solutions.\n",
       "\n",
       "Second, you can run a private competition for internal competition organizers only (competitors will need to be invited). This is ideal if you want to create a closed environment for your internal competition.\n",
       "\n",
       "## Create a private competition\n",
       "\n",
       "Follow these steps:\n",
       "\n",
       "- [Create a new notebook](https://www.kaggle.com/docs/notebooks#creating-a-new-solution) in your competition folder\n",
       "- [Add your solution to the leaderboard](https://www.kaggle.com/competition/leaderboards#create-a-new-leaderboard-in-a-private-competition-folder)\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Generate Prompt using template\n",
    "prompt = template.format(\n",
    "    Category=\"kaggle-competitions\",\n",
    "    Question=\"How you can create a in-class competition?\",\n",
    "    Answer=\"\"\n",
    ")\n",
    "\n",
    "# Infer\n",
    "output = gemma_lm.generate(prompt, max_length=256)\n",
    "\n",
    "# Colorize\n",
    "output = colorize_text(output)\n",
    "\n",
    "# Display in markdown\n",
    "display(Markdown(output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "76648236",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-31T10:41:54.867818Z",
     "iopub.status.busy": "2024-03-31T10:41:54.867494Z",
     "iopub.status.idle": "2024-03-31T10:42:01.170533Z",
     "shell.execute_reply": "2024-03-31T10:42:01.169505Z"
    },
    "papermill": {
     "duration": 6.396189,
     "end_time": "2024-03-31T10:42:01.172812",
     "exception": false,
     "start_time": "2024-03-31T10:41:54.776623",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='blue'>Category:</font>**\n",
       "kaggle-kaggle-competitions\n",
       "\n",
       "**<font color='red'>Question:</font>**\n",
       "How you can create a in-class competition?\n",
       "\n",
       "**<font color='green'>Answer:</font>**\n",
       "You can create a competition for attendees to a live workshop. This is a great way to drive engagement and participation in your event.\n",
       "\n",
       "## Creating a Workshop-Type Competition\n",
       "\n",
       "## Step 1: Set Up the Competition Settings\n",
       "\n",
       "Navigate to the \"Settings\" tab for the workshop you want to create a competition for.\n",
       "\n",
       "In the \"Format\" menu, select \"Workshop\" as the \"Competition Format\".\n",
       "\n",
       "In the \"Dataset\" menu, select \"Custom\".\n",
       "\n",
       "In the \"Data\" section, upload your dataset (if you haven't done so already).\n",
       "\n",
       "In the \"Scoring\" section, select the metric you want to use to determine the winner(s) of the competition.\n",
       "\n",
       "You may choose between the following options:\n",
       "\n",
       "- \"Submission Accuracy\": If you’re hosting an accuracy-type competition and want your participants to submit machine learning models, this is the metric for them! Submission accuracy is the proportion of predictions by the model that are correct.\n",
       "- \"Submission Macro F1\": If you’re hosting an F1-type competition and want your participants to submit predictions, this"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Generate Prompt using template\n",
    "prompt = template.format(\n",
    "    Category=\"kaggle-competitions\",\n",
    "    Question=\"How you can create a in-class competition?\",\n",
    "    Answer=\"\"\n",
    ")\n",
    "\n",
    "# Infer\n",
    "output = gemma_lm.generate(prompt, max_length=256)\n",
    "\n",
    "# Colorize\n",
    "output = colorize_text(output)\n",
    "\n",
    "# Display in markdown\n",
    "display(Markdown(output))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f7aa2f9",
   "metadata": {
    "papermill": {
     "duration": 0.090122,
     "end_time": "2024-03-31T10:42:01.355280",
     "exception": false,
     "start_time": "2024-03-31T10:42:01.265158",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Conclusion\n",
    "\n",
    "The result is not bad, especially compared to the model without fine-tuning. Though it's not exactly what we're looking for, it's important to remember that we only fine-tuned this model using $60$ samples without any augmentation or advanced prompting. Therefore, there is ample room for improvement. Here are some tips to improve performance:\n",
    "\n",
    "- Try using the larger version of **Gemma** (7B).\n",
    "- Increase `sequence_length`.\n",
    "- Experiment with advanced prompt engineering techniques.\n",
    "- Implement augmentation to increase the number of samples.\n",
    "- Utilize a learning rate scheduler."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bfb58c6",
   "metadata": {
    "papermill": {
     "duration": 0.088662,
     "end_time": "2024-03-31T10:42:01.533392",
     "exception": false,
     "start_time": "2024-03-31T10:42:01.444730",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Reference\n",
    "* [Fine-tune Gemma models in Keras using LoRA](https://www.kaggle.com/code/nilaychauhan/fine-tune-gemma-models-in-keras-using-lora)\n",
    "* [Parameter-efficient fine-tuning of GPT-2 with LoRA](https://keras.io/examples/nlp/parameter_efficient_finetuning_of_gpt2_with_lora/)\n",
    "* [Gemma - KerasNLP](https://keras.io/api/keras_nlp/models/gemma/)"
   ]
  }
 ],
 "metadata": {
  "kaggle": {
   "accelerator": "gpu",
   "dataSources": [
    {
     "databundleVersionId": 7669720,
     "sourceId": 64148,
     "sourceType": "competition"
    },
    {
     "datasetId": 4484051,
     "sourceId": 7711309,
     "sourceType": "datasetVersion"
    },
    {
     "modelInstanceId": 5171,
     "sourceId": 11371,
     "sourceType": "modelInstanceVersion"
    }
   ],
   "dockerImageVersionId": 30648,
   "isGpuEnabled": true,
   "isInternetEnabled": true,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 937.56924,
   "end_time": "2024-03-31T10:42:05.224546",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2024-03-31T10:26:27.655306",
   "version": "2.5.0"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "0273425219354c10a663da423a9b53a0": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "3567a147ae024f208d67db5ec9a70b74": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "38abc7f186204b7bb5fa01fb9e5858bc": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_82624678224045f9b3eb9d50ddbf2e53",
        "IPY_MODEL_dde999bbf11a44b7a07922f156885d2e",
        "IPY_MODEL_8f5cff8d2af64cf38c82f8ecc3544e01"
       ],
       "layout": "IPY_MODEL_bfbb26c9083c4862b91a75f5b3cbdc52"
      }
     },
     "5cbfed354a5b4b6bb936047c0f78ea30": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "63c35c32087f41409c97db492a814b01": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "82624678224045f9b3eb9d50ddbf2e53": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_3567a147ae024f208d67db5ec9a70b74",
       "placeholder": "​",
       "style": "IPY_MODEL_63c35c32087f41409c97db492a814b01",
       "value": "100%"
      }
     },
     "8f5cff8d2af64cf38c82f8ecc3544e01": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_5cbfed354a5b4b6bb936047c0f78ea30",
       "placeholder": "​",
       "style": "IPY_MODEL_b7235bc32da24849a3fc165e2ca1852e",
       "value": " 60/60 [00:00&lt;00:00, 4120.61it/s]"
      }
     },
     "b7235bc32da24849a3fc165e2ca1852e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "bfbb26c9083c4862b91a75f5b3cbdc52": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "dde999bbf11a44b7a07922f156885d2e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_0273425219354c10a663da423a9b53a0",
       "max": 60.0,
       "min": 0.0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_f0bec041febf454c86d325883444adef",
       "value": 60.0
      }
     },
     "f0bec041febf454c86d325883444adef": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}