03_M01L02_Lab_Creating_Custom_Layers_and_Models__M.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/BQm_EV6i0_j80CQZ8vcLvw/SN-web-lightmode.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Lab: Creating Custom Layers and Models**\n",
    "\n",
    "###### Estimated time needed:  30 minutes  \n",
    "\n",
    "In this lab, you will learn to create custom layers and integrate them into a Keras model. You will compile, train, and evaluate the model. \n",
    "\n",
    "##### Learning objectives \n",
    "\n",
    "By the end of this lab, you will: \n",
    "- Create custom layers and integrate them into a Keras model \n",
    "- Compile, train, and evaluate the model \n",
    "\n",
    "##### Prerequisites:\n",
    "- Basic understanding of Python and Keras. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Steps \n",
    "\n",
    "**Step 1: Import libraries**\n",
    "\n",
    "Before you start, import the required libraries: TensorFlow and Keras. Keras is included within TensorFlow as `tensorflow.keras`. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tensorflow==2.16.2 in /opt/conda/lib/python3.11/site-packages (2.16.2)\n",
      "Requirement already satisfied: absl-py>=1.0.0 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (2.1.0)\n",
      "Requirement already satisfied: astunparse>=1.6.0 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (1.6.3)\n",
      "Requirement already satisfied: flatbuffers>=23.5.26 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (24.3.25)\n",
      "Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (0.6.0)\n",
      "Requirement already satisfied: google-pasta>=0.1.1 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (0.2.0)\n",
      "Requirement already satisfied: h5py>=3.10.0 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (3.12.1)\n",
      "Requirement already satisfied: libclang>=13.0.0 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (18.1.1)\n",
      "Requirement already satisfied: ml-dtypes~=0.3.1 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (0.3.2)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (3.4.0)\n",
      "Requirement already satisfied: packaging in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (24.0)\n",
      "Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (4.25.5)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (2.31.0)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (69.5.1)\n",
      "Requirement already satisfied: six>=1.12.0 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (1.16.0)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (2.5.0)\n",
      "Requirement already satisfied: typing-extensions>=3.6.6 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (4.12.2)\n",
      "Requirement already satisfied: wrapt>=1.11.0 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (1.17.0)\n",
      "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (1.68.0)\n",
      "Requirement already satisfied: tensorboard<2.17,>=2.16 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (2.16.2)\n",
      "Requirement already satisfied: keras>=3.0.0 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (3.7.0)\n",
      "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (0.37.1)\n",
      "Requirement already satisfied: numpy<2.0.0,>=1.23.5 in /opt/conda/lib/python3.11/site-packages (from tensorflow==2.16.2) (1.26.4)\n",
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in /opt/conda/lib/python3.11/site-packages (from astunparse>=1.6.0->tensorflow==2.16.2) (0.43.0)\n",
      "Requirement already satisfied: rich in /opt/conda/lib/python3.11/site-packages (from keras>=3.0.0->tensorflow==2.16.2) (13.9.4)\n",
      "Requirement already satisfied: namex in /opt/conda/lib/python3.11/site-packages (from keras>=3.0.0->tensorflow==2.16.2) (0.0.8)\n",
      "Requirement already satisfied: optree in /opt/conda/lib/python3.11/site-packages (from keras>=3.0.0->tensorflow==2.16.2) (0.13.1)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests<3,>=2.21.0->tensorflow==2.16.2) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests<3,>=2.21.0->tensorflow==2.16.2) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.11/site-packages (from requests<3,>=2.21.0->tensorflow==2.16.2) (2.2.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests<3,>=2.21.0->tensorflow==2.16.2) (2024.8.30)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.11/site-packages (from tensorboard<2.17,>=2.16->tensorflow==2.16.2) (3.7)\n",
      "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /opt/conda/lib/python3.11/site-packages (from tensorboard<2.17,>=2.16->tensorflow==2.16.2) (0.7.2)\n",
      "Requirement already satisfied: werkzeug>=1.0.1 in /opt/conda/lib/python3.11/site-packages (from tensorboard<2.17,>=2.16->tensorflow==2.16.2) (3.1.3)\n",
      "Requirement already satisfied: MarkupSafe>=2.1.1 in /opt/conda/lib/python3.11/site-packages (from werkzeug>=1.0.1->tensorboard<2.17,>=2.16->tensorflow==2.16.2) (2.1.5)\n",
      "Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/conda/lib/python3.11/site-packages (from rich->keras>=3.0.0->tensorflow==2.16.2) (3.0.0)\n",
      "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/conda/lib/python3.11/site-packages (from rich->keras>=3.0.0->tensorflow==2.16.2) (2.18.0)\n",
      "Requirement already satisfied: mdurl~=0.1 in /opt/conda/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich->keras>=3.0.0->tensorflow==2.16.2) (0.1.2)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-28 10:13:30.922467: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n",
      "2024-11-28 10:13:30.924722: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2024-11-28 10:13:30.928510: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2024-11-28 10:13:30.942275: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:479] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2024-11-28 10:13:30.968722: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:10575] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2024-11-28 10:13:30.968770: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1442] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2024-11-28 10:13:30.986405: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2024-11-28 10:13:32.155923: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
     ]
    }
   ],
   "source": [
    "!pip install tensorflow==2.16.2\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Layer\n",
    "from tensorflow.keras.models import Sequential"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 2: Define a custom layer**\n",
    "\n",
    "Define a custom dense layer with 32 units and ReLU activation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomDenseLayer(Layer):\n",
    "    def __init__(self, units=32):\n",
    "        super(CustomDenseLayer, self).__init__()\n",
    "        self.units = units\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        self.w = self.add_weight(shape=(input_shape[-1], self.units),\n",
    "                                 initializer='random_normal',\n",
    "                                 trainable=True)\n",
    "        self.b = self.add_weight(shape=(self.units,),\n",
    "                                 initializer='zeros',\n",
    "                                 trainable=True)\n",
    "    def call(self, inputs):\n",
    "        return tf.nn.relu(tf.matmul(inputs, self.w) + self.b)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 3: Integrate the custom layer into a model**\n",
    "\n",
    "Create a Keras model using the custom layer. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Softmax\n",
    "\n",
    "# Define the model with Softmax in the output layer\n",
    "model = Sequential([\n",
    "    CustomDenseLayer(128),\n",
    "    CustomDenseLayer(10),  # Hidden layer with ReLU activation\n",
    "    Softmax()              # Output layer with Softmax activation for multi-class classification\n",
    "])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **Softmax** activation function is used in the output layer for multi-class classification tasks, ensuring the model outputs probabilities that sum up to 1 for each class, which aligns with categorical cross-entropy as the loss function. This adjustment ensures the model is optimized correctly for multi-class classification.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 4: Compile the model**\n",
    "\n",
    "Compile the model with the Adam optimizer and categorical cross-entropy loss. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model summary before building:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"sequential\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"sequential\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)                    </span>┃<span style=\"font-weight: bold\"> Output Shape           </span>┃<span style=\"font-weight: bold\">       Param # </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ custom_dense_layer              │ ?                      │   <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (unbuilt) │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">CustomDenseLayer</span>)              │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ custom_dense_layer_1            │ ?                      │   <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (unbuilt) │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">CustomDenseLayer</span>)              │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ softmax (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Softmax</span>)               │ ?                      │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      Param #\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ custom_dense_layer              │ ?                      │   \u001b[38;5;34m0\u001b[0m (unbuilt) │\n",
       "│ (\u001b[38;5;33mCustomDenseLayer\u001b[0m)              │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ custom_dense_layer_1            │ ?                      │   \u001b[38;5;34m0\u001b[0m (unbuilt) │\n",
       "│ (\u001b[38;5;33mCustomDenseLayer\u001b[0m)              │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ softmax (\u001b[38;5;33mSoftmax\u001b[0m)               │ ?                      │             \u001b[38;5;34m0\u001b[0m │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Model summary after building:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"sequential\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"sequential\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)                    </span>┃<span style=\"font-weight: bold\"> Output Shape           </span>┃<span style=\"font-weight: bold\">       Param # </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ custom_dense_layer              │ (<span style=\"color: #00af00; text-decoration-color: #00af00\">1000</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">128</span>)            │         <span style=\"color: #00af00; text-decoration-color: #00af00\">2,688</span> │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">CustomDenseLayer</span>)              │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ custom_dense_layer_1            │ (<span style=\"color: #00af00; text-decoration-color: #00af00\">1000</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">10</span>)             │         <span style=\"color: #00af00; text-decoration-color: #00af00\">1,290</span> │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">CustomDenseLayer</span>)              │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ softmax (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Softmax</span>)               │ (<span style=\"color: #00af00; text-decoration-color: #00af00\">1000</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">10</span>)             │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      Param #\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ custom_dense_layer              │ (\u001b[38;5;34m1000\u001b[0m, \u001b[38;5;34m128\u001b[0m)            │         \u001b[38;5;34m2,688\u001b[0m │\n",
       "│ (\u001b[38;5;33mCustomDenseLayer\u001b[0m)              │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ custom_dense_layer_1            │ (\u001b[38;5;34m1000\u001b[0m, \u001b[38;5;34m10\u001b[0m)             │         \u001b[38;5;34m1,290\u001b[0m │\n",
       "│ (\u001b[38;5;33mCustomDenseLayer\u001b[0m)              │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ softmax (\u001b[38;5;33mSoftmax\u001b[0m)               │ (\u001b[38;5;34m1000\u001b[0m, \u001b[38;5;34m10\u001b[0m)             │             \u001b[38;5;34m0\u001b[0m │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">3,978</span> (15.54 KB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m3,978\u001b[0m (15.54 KB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">3,978</span> (15.54 KB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m3,978\u001b[0m (15.54 KB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.compile(optimizer='adam', loss='categorical_crossentropy')\n",
    "print(\"Model summary before building:\")\n",
    "model.summary()\n",
    "\n",
    "# Build the model to show parameters\n",
    "model.build((1000, 20))\n",
    "print(\"\\nModel summary after building:\")\n",
    "model.summary()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 5: Train the model**\n",
    "\n",
    "Train the model on some example data. For this example, you will generate random data for training. In practice, use a real data set. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 4ms/step - loss: 2.3065  \n",
      "Epoch 2/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step - loss: 2.3013\n",
      "Epoch 3/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step - loss: 2.3033\n",
      "Epoch 4/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 2.3004\n",
      "Epoch 5/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 2.2953\n",
      "Epoch 6/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 2.2943\n",
      "Epoch 7/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 2.2922\n",
      "Epoch 8/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 2.2966\n",
      "Epoch 9/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 2.2928\n",
      "Epoch 10/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 2.2897\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.history.History at 0x7f762c769790>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np \n",
    "\n",
    "# Generate random data \n",
    "x_train = np.random.random((1000, 20)) \n",
    "y_train = np.random.randint(10, size=(1000, 1)) \n",
    "\n",
    "# Convert labels to categorical one-hot encoding \n",
    "y_train = tf.keras.utils.to_categorical(y_train, num_classes=10) \n",
    "model.fit(x_train, y_train, epochs=10, batch_size=32) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 6: Evaluate the model**\n",
    "\n",
    "Evaluate the model using test data to see its performance. \n",
    "\n",
    "For this example, you will generate random test data. In practice, use a real data set. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m7/7\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 6ms/step - loss: 2.2975  \n",
      "Test loss: 2.2974681854248047\n"
     ]
    }
   ],
   "source": [
    "# Generate random test data \n",
    "x_test = np.random.random((200, 20)) \n",
    "y_test = np.random.randint(10, size=(200, 1)) \n",
    "\n",
    "# Convert labels to categorical one-hot encoding \n",
    "y_test = tf.keras.utils.to_categorical(y_test, num_classes=10) \n",
    "\n",
    "# Evaluate the model \n",
    "loss = model.evaluate(x_test, y_test) \n",
    "print(f'Test loss: {loss}') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 1: Visualize Model Architecture\n",
    "\n",
    "**Objective:** Visualize the architecture of the custom Keras model to understand its structure.\n",
    "\n",
    "**Instructions:**\n",
    "1. Use the `plot_model` function from `tensorflow.keras.utils` to visualize the model architecture.\n",
    "2. Save the plot as an image file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pydot\n",
      "  Downloading pydot-3.0.2-py3-none-any.whl.metadata (10 kB)\n",
      "Collecting graphviz\n",
      "  Downloading graphviz-0.20.3-py3-none-any.whl.metadata (12 kB)\n",
      "Collecting pyparsing>=3.0.9 (from pydot)\n",
      "  Downloading pyparsing-3.2.0-py3-none-any.whl.metadata (5.0 kB)\n",
      "Downloading pydot-3.0.2-py3-none-any.whl (35 kB)\n",
      "Downloading graphviz-0.20.3-py3-none-any.whl (47 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m47.1/47.1 kB\u001b[0m \u001b[31m6.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading pyparsing-3.2.0-py3-none-any.whl (106 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m106.9/106.9 kB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: pyparsing, graphviz, pydot\n",
      "Successfully installed graphviz-0.20.3 pydot-3.0.2 pyparsing-3.2.0\n",
      "You must install pydot (`pip install pydot`) for `plot_model` to work.\n"
     ]
    }
   ],
   "source": [
    "!pip install pydot graphviz\n",
    "\n",
    "from tensorflow.keras.utils import plot_model\n",
    "\n",
    "# Visualize the model architecture\n",
    "plot_model(model, to_file='model_architecture.png', show_shapes=True, show_layer_names=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "\n",
    "!pip install pydot graphviz\n",
    "\n",
    "from tensorflow.keras.utils import plot_model\n",
    "\n",
    "# Visualize the model architecture\n",
    "plot_model(model, to_file='model_architecture.png', show_shapes=True, show_layer_names=True)\n",
    "\n",
    "\n",
    " ```   \n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 2: Add Dropout Layer\n",
    "\n",
    "**Objective:** Enhance the model by adding a Dropout layer to prevent overfitting.\n",
    "\n",
    "**Instructions:**\n",
    "1. Add a Dropout layer between the custom dense layers.\n",
    "2. Recompile the model and observe the impact on training.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 5ms/step - loss: 7.9745 \n",
      "Epoch 2/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step - loss: 4.0088\n",
      "Epoch 3/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 5ms/step - loss: 2.8161\n",
      "Epoch 4/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step - loss: 2.4266\n",
      "Epoch 5/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step - loss: 2.4451\n",
      "Epoch 6/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 2.4288\n",
      "Epoch 7/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step - loss: 2.3263\n",
      "Epoch 8/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 2.3653\n",
      "Epoch 9/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 2.3788\n",
      "Epoch 10/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step - loss: 2.3739\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.history.History at 0x7f762c578a50>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow.keras.layers import Dropout\n",
    "\n",
    "# Modify the model to include a Dropout layer\n",
    "model = Sequential([\n",
    "    CustomDenseLayer(64),\n",
    "    Dropout(0.5),\n",
    "    CustomDenseLayer(10)\n",
    "])\n",
    "\n",
    "# Recompile the model\n",
    "model.compile(optimizer='adam', loss='categorical_crossentropy')\n",
    "\n",
    "# Train the model again\n",
    "model.fit(x_train, y_train, epochs=10, batch_size=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "\n",
    "from tensorflow.keras.layers import Dropout\n",
    "\n",
    "# Modify the model to include a Dropout layer\n",
    "model = Sequential([\n",
    "    CustomDenseLayer(64),\n",
    "    Dropout(0.5),\n",
    "    CustomDenseLayer(10)\n",
    "])\n",
    "\n",
    "# Recompile the model\n",
    "model.compile(optimizer='adam', loss='categorical_crossentropy')\n",
    "\n",
    "# Train the model again\n",
    "model.fit(x_train, y_train, epochs=10, batch_size=32)\n",
    " ```   \n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 3: Adjust the Number of Units in Custom Layer\n",
    "\n",
    "**Objective:** Experiment with different numbers of units in the custom dense layer to observe the impact on performance.\n",
    "\n",
    "**Instructions:**\n",
    "1. Change the number of units in the `CustomDenseLayer` to 128.\n",
    "2. Recompile, train, and evaluate the model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 4ms/step - loss: 4.1158  \n",
      "Epoch 2/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step - loss: 3.6579 \n",
      "Epoch 3/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 5ms/step - loss: 3.5760\n",
      "Epoch 4/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step - loss: 3.8406\n",
      "Epoch 5/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step - loss: 3.7006\n",
      "Epoch 6/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 3.7714\n",
      "Epoch 7/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 3.6474\n",
      "Epoch 8/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 3.8713\n",
      "Epoch 9/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 3.6308\n",
      "Epoch 10/10\n",
      "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step - loss: 3.6689\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.history.History at 0x7f760de14a10>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define a custom layer with 128 units\n",
    "class CustomDenseLayer(Layer):\n",
    "    def __init__(self, units=128):\n",
    "        super(CustomDenseLayer, self).__init__()\n",
    "        self.units = units\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        self.w = self.add_weight(shape=(input_shape[-1], self.units),\n",
    "                                 initializer='random_normal',\n",
    "                                 trainable=True)\n",
    "        self.b = self.add_weight(shape=(self.units,),\n",
    "                                 initializer='zeros',\n",
    "                                 trainable=True)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        return tf.nn.relu(tf.matmul(inputs, self.w) + self.b)\n",
    "\n",
    "# Integrate the new custom layer into a model\n",
    "model = Sequential([\n",
    "    CustomDenseLayer(128),\n",
    "    CustomDenseLayer(10)\n",
    "])\n",
    "\n",
    "# Recompile the model\n",
    "model.compile(optimizer='adam', loss='categorical_crossentropy')\n",
    "\n",
    "# Train the model again\n",
    "model.fit(x_train, y_train, epochs=10, batch_size=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "\n",
    "# Define a custom layer with 128 units\n",
    "class CustomDenseLayer(Layer):\n",
    "    def __init__(self, units=128):\n",
    "        super(CustomDenseLayer, self).__init__()\n",
    "        self.units = units\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        self.w = self.add_weight(shape=(input_shape[-1], self.units),\n",
    "                                 initializer='random_normal',\n",
    "                                 trainable=True)\n",
    "        self.b = self.add_weight(shape=(self.units,),\n",
    "                                 initializer='zeros',\n",
    "                                 trainable=True)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        return tf.nn.relu(tf.matmul(inputs, self.w) + self.b)\n",
    "\n",
    "# Integrate the new custom layer into a model\n",
    "model = Sequential([\n",
    "    CustomDenseLayer(128),\n",
    "    CustomDenseLayer(10)\n",
    "])\n",
    "\n",
    "# Recompile the model\n",
    "model.compile(optimizer='adam', loss='categorical_crossentropy')\n",
    "\n",
    "# Train the model again\n",
    "model.fit(x_train, y_train, epochs=10, batch_size=32)\n",
    " ```   \n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "\n",
    "By completing these exercises, students will:\n",
    "\n",
    "1. Visualize the architecture of their custom Keras model.\n",
    "2. Understand the impact of adding Dropout layers to prevent overfitting.\n",
    "3. Experiment with different configurations of the custom dense layer to observe performance changes.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion \n",
    "\n",
    "Congratulations! You have successfully created and trained a custom layer in Keras. This lab exercise demonstrated how to extend Keras’s capabilities by creating custom layers and integrating them into a model. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid character '©' (U+00A9) (1701490070.py, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  Cell \u001b[0;32mIn[10], line 1\u001b[0;36m\u001b[0m\n\u001b[0;31m    Copyright © IBM Corporation. All rights reserved.\u001b[0m\n\u001b[0m              ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid character '©' (U+00A9)\n"
     ]
    }
   ],
   "source": [
    "Copyright © IBM Corporation. All rights reserved."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "prev_pub_hash": "549280a90960ac2b7510a6dd71e9ce639bcd2ccae4a88ef2f05039b753eda708"
 },
 "nbformat": 4,
 "nbformat_minor": 4
}