NNFromScratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0e9128f2",
   "metadata": {},
   "source": [
    "### Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b43cf990",
   "metadata": {},
   "source": [
    "The goal of this notebook is to build from scratch a neural network for an image classficiation task.\n",
    "\n",
    "This is to have a good grasp of deep learning concepts .."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c44e778",
   "metadata": {},
   "source": [
    "## Plan"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dccca0e2",
   "metadata": {},
   "source": [
    "**1-** Prepare the dataset\n",
    "\n",
    "**2-** Implementing the forward pass\n",
    "\n",
    "**3-** Implementing the backwad pass\n",
    "\n",
    "**4-** Evaluating the performance of the nn\n",
    "\n",
    "**5-** Experiments with the nn parametres and how they affect the performances \n",
    "\n",
    "Some questions that we want to answer are : \n",
    "\n",
    "* Can we find some correlation between the variance in the dataset and the size of the neural network? \n",
    "* Is the bias part of the neurons important in the task?\n",
    "* Does the activation function affects the value to be choosen for the learning rate\n",
    "\n",
    "New questions: \n",
    "* Why CCE makes the NN converges faster than MSE? Why does it make it more accurate?\n",
    "* Which optimization algo to choose (adam, rmsprop, sgd ...) and why?\n",
    "* How does the weight init affects the learning of the nn? "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0e8dec1",
   "metadata": {},
   "source": [
    "## 1-  Loading the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6fe544e9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shape: (60000, 28, 28)\n",
      "Training labels shape: (60000,)\n",
      "Test data shape: (10000, 28, 28)\n",
      "Test labels shape: (10000,)\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# Load the MNIST dataset                                \n",
    "mnist = tf.keras.datasets.mnist\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "\n",
    "# Shuffle the training data\n",
    "indices = tf.range(start=0, limit=tf.shape(x_train)[0], dtype=tf.int32)\n",
    "shuffled_indices = tf.random.shuffle(indices)\n",
    "\n",
    "x_train = tf.gather(x_train, shuffled_indices)\n",
    "y_train = tf.gather(y_train, shuffled_indices)\n",
    "\n",
    "# Convert tensors back to numpy arrays\n",
    "x_train = x_train.numpy()\n",
    "y_train = y_train.numpy()\n",
    "\n",
    "# Display the shape of the loaded data\n",
    "print(\"Training data shape:\", x_train.shape)  # (60000, 28, 28)\n",
    "print(\"Training labels shape:\", y_train.shape)  # (60000,)\n",
    "print(\"Test data shape:\", x_test.shape)  # (10000, 28, 28)\n",
    "print(\"Test labels shape:\", y_test.shape)  # (10000,)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41459d3c",
   "metadata": {},
   "source": [
    "##  2-  Visualizing the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9642a8f9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAAD3CAYAAADmIkO7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAHOklEQVR4nO3dT4hW1x3H4XPUZgwRR8lCkUAlhEI3IQR12wi6aCEWBpcaDQjixpWQgGYh2blwI1gCEyYI7aJQEDdKCFkKgSAiNBtpqBSDfxISFB1E4+2q0ID3jJl3xvf73nme5fy47z0MfDzi8d63dl1XgDyrxr0A4NnECaHECaHECaHECaHECaHECaHEOQC11qla66e11hu11vu11qu11j+Oe12MRpzDsKaU8p9Syh9KKdOllBOllL/XWreOc1GMpvofQsNUa71WSjnZdd0/xr0WFsfOOUC11k2llN+VUv457rWweHbOgam1/qaUcrGU8q+u6w6Pez0snjgHpNa6qpTyt1LK+lLKn7uuezzmJTGCNeNeAEuj1lpLKZ+WUjaVUv4kzMknzuH4Synl96WUXV3XzY97MYzOX2sHoNb621LKv0spj0opT/5vdLjrur+OZVGMTJwQylEKhBInhBInhBInhGoepdRa/WsRLLOu6+qzfm7nhFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDNrwDk2dasaf/aNm/e3JwfPny4d/baa681rz148GBzPqrZ2dne2fHjx5vX3rlzZ6mXs6LZOSGUOCGUOCGUOCGUOCGUOCGUOCFU7bquf1hr/3AFO3fuXHO+b9++F7SSF2tubq45/+ijj5rz7777bimXMxhd19Vn/dzOCaHECaHECaHECaHECaHECaHECaE8z8lze//995vzhZ5zPXDgwFIuZ/DsnBBKnBBKnBBKnBBKnBBKnBDKUcoiXLx4sTlf6JGxx48f986+//77Ra3pea1du7Y537hx46I/++WXX27OV69e3Zz//PPPi773ENk5IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZRXYy7C9PR0c753797m/IcffuidnT9/fjFLem47d+5szltnuC+99NJI937rrbea82vXro30+ZPKqzFhwogTQokTQokTQokTQokTQokTQjnn5BeuXLnSO1vonHIhp06das4//PDDkT5/UjnnhAkjTgglTgglTgglTgglTgglTgjlvbUrzNtvv92cb926ddnuffbs2WX77CGyc0IocUIocUIocUIocUIocUIocUIo55wrzNGjR5vzDRs2vJiFsCA7J4QSJ4QSJ4QSJ4QSJ4QSJ4RylDIw7777bnO+Z8+eZbv3l19+2ZzfvXt32e49RHZOCCVOCCVOCCVOCCVOCCVOCCVOCOWcc8Is9EjXxx9/PNL1o7h8+XJzPj8/v2z3HiI7J4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4RyzhlmoXPIubm55vzNN99cwtX80u3bt5vzTz75ZNnuvRLZOSGUOCGUOCGUOCGUOCGUOCGUOCGUc84x2LZtW+/sxIkTzWuX872zpZRy69at3tnMzEzz2ps3by71clY0OyeEEieEEieEEieEEieEEieEEieEcs65COvWrWvOd+/e3Zx/8MEHvbMdO3Ysak1L5caNG72zb7755gWuBDsnhBInhBInhBInhBInhBInhKpd1/UPa+0fTrB33nmnOT9z5kxzPjU11Zy/8cYbv3ZJE+HChQvN+f79+5vz+/fvL+VyBqPruvqsn9s5IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IdRgzzk3btzYO/v222+b105PTy/1claE1uNmpZRy5MiR5vzSpUtLuZyJ4ZwTJow4IZQ4IZQ4IZQ4IZQ4IZQ4IdRgzzl37drVO/v8889f4Er4nwcPHjTnr7/+eu/s7t27S72cGM45YcKIE0KJE0KJE0KJE0KJE0KJE0IN9isAr1+/Pu4lLIubN2825wu9W3ZmZqY537Rp069e0/N65ZVXmvNjx471zlpfmzhUdk4IJU4IJU4IJU4IJU4IJU4INdhHxl599dXe2aiPH/3000/N+Weffdac37p1q3c2OzvbvPbJkyfN+b1795rz9evXN+dr1vSfrp0+fbp57XvvvdecL+TRo0e9sy1btjSv/fHHH0e69zh5ZAwmjDghlDghlDghlDghlDghlDgh1GAfGZufn++dXb16tXlt6xyylFJOnjzZnH/11VfN+TgtdA7acujQoeZ81ar2n/X79u1rzqempnpntT7zKHDQ7JwQSpwQSpwQSpwQSpwQSpwQSpwQarDnnA8fPuydbd++vXlt6xnXUkp5+vTpotY06RZ6lvSLL75ozhc65/z66697Z61z66Gyc0IocUIocUIocUIocUIocUIocUKowb63FiaF99bChBEnhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhGp+BSAwPnZOCCVOCCVOCCVOCCVOCCVOCPVfGWM+6h2PFHIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "# Fixing the seed to assure reproducibility\n",
    "seed_value = 42\n",
    "np.random.seed(seed_value)\n",
    "\n",
    "\n",
    "random_idx = np.random.randint(1000)\n",
    "\n",
    "image = x_train[random_idx]\n",
    "label = y_train[random_idx]\n",
    "\n",
    "\n",
    "def visualize_image(image, label):\n",
    "    plt.imshow(image, cmap='gray')  # Display the image\n",
    "    plt.axis('off')  # Turn off the axes\n",
    "    plt.title(label)\n",
    "    plt.show()  # Show the image\n",
    "    \n",
    "visualize_image(image, label)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "026407de",
   "metadata": {},
   "source": [
    "==> Each image is a **28*28** pixels, which gives a list of **784** elements"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1adace72",
   "metadata": {},
   "source": [
    "## 3- Implementing the forward pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4bd6c35",
   "metadata": {},
   "source": [
    "Our neural network, would be such that there is only one hidden layer with **300** neurons, and the output layer with **10**neurons\n",
    "\n",
    "For now, the activations functions would be all linear, and in to generate the output we take the argmax.\n",
    "\n",
    "Our loss function will be the Mean Squared Error, aka **MSE**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "27c41a1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_layer = image.flatten()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "b2a15531",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "784\n"
     ]
    }
   ],
   "source": [
    "## Initializing the weights linking the input layer with the hidden layer\n",
    "\n",
    "SIZE_INPUT_LAYER = len(input_layer) #784\n",
    "SIZE_HIDDEN_LAYER = 289\n",
    "SIZE_OUTPUT_LAYER = 10\n",
    "print(len(input_layer))\n",
    "\n",
    "def init_weights():\n",
    "    weight_input_hidden = np.random.random((SIZE_INPUT_LAYER, SIZE_HIDDEN_LAYER))\n",
    "    weight_hidden_output = np.random.random((SIZE_HIDDEN_LAYER, SIZE_OUTPUT_LAYER))\n",
    "    \n",
    "    return weight_input_hidden, weight_hidden_output\n",
    "\n",
    "weight_input_hidden, weight_hidden_output = init_weights()\n",
    "weights = weight_input_hidden, weight_hidden_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7dc22bac",
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_weights_he():\n",
    "    # He initialization for weight_input_hidden\n",
    "    limit_input_hidden = np.sqrt(2 / SIZE_INPUT_LAYER)\n",
    "    weight_input_hidden = np.random.randn(SIZE_INPUT_LAYER, SIZE_HIDDEN_LAYER) * limit_input_hidden\n",
    "    \n",
    "    # He initialization for weight_hidden_output\n",
    "    limit_hidden_output = np.sqrt(2 / SIZE_HIDDEN_LAYER)\n",
    "    weight_hidden_output = np.random.randn(SIZE_HIDDEN_LAYER, SIZE_OUTPUT_LAYER) * limit_hidden_output\n",
    "    \n",
    "    return weight_input_hidden, weight_hidden_output\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d66bfa5a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((784, 289), (289, 10))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def init_weights_xavier():\n",
    "    # Xavier initialization for weight_input_hidden\n",
    "    limit_input_hidden = np.sqrt(6 / (size_input_layer + size_hidden_layer))\n",
    "    weight_input_hidden = np.random.uniform(-limit_input_hidden, limit_input_hidden, \n",
    "                                            (size_input_layer, size_hidden_layer))\n",
    "    \n",
    "    # Xavier initialization for weight_hidden_output\n",
    "    limit_hidden_output = np.sqrt(6 / (size_hidden_layer + size_output_layer))\n",
    "    weight_hidden_output = np.random.uniform(-limit_hidden_output, limit_hidden_output, \n",
    "                                             (size_hidden_layer, size_output_layer))\n",
    "    \n",
    "    return weight_input_hidden, weight_hidden_output\n",
    "\n",
    "# Test the initialization\n",
    "weights_xavier = init_weights_xavier()\n",
    "weights_xavier[0].shape, weights_xavier[1].shape  # Just to check if the shapes are correct\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "96a69d14",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hidden layer result\n",
    "hidden_representation = np.dot(input_layer, weight_input_hidden)\n",
    "\n",
    "# Min max normalization\n",
    "mi = hidden_representation.min()\n",
    "ma = hidden_representation.max()\n",
    "\n",
    "# min max normalization\n",
    "norm_hidden_representation = (hidden_representation - mi)/(ma-mi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "27f1cfdb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1d7ac740670>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAUEElEQVR4nO3de5BUdXYH8O9XhscwDK8hAs5QgJSirgS0BopVJBhfQFS0sAo1JLiuZW0SjaZUihUqbuUvZVcTY9YHKgkJKsQXS22BC0ERLAQFIi9BGIHIADOgsgMOD3mc/NEXq3fsHubce7sZ9/f9VE1Nz9xz5vy6m0O/7r2HZgYR+eN3ztlegIgUh5pdJBBqdpFAqNlFAqFmFwlESTGLlZeXW0VFhSvn8OHD7jplZWWu+EOHDrlrlJeXu3MaGxvdOceOHXPntG/f3p3Tpk0bV3zXrl3dNXbv3u3OiXM7//73v3fnDBw40BVfV1fnrhHnvmzXrp0rvqGhAYcPH2aubUVt9oqKCkydOtWVs379ened6upqV/zSpUvdNa6++mp3zocffujO2blzpzunf//+7pzOnTu74sePH++u4b3vAWDEiBHunPnz57tzli9f7oqfPn26u0ZNTY07p7Ky0hU/a9asvNv0NF4kEGp2kUAkanaSo0l+RrKG5JS0FiUi6Yvd7CTbAPg1gDEALgFwB8lL0lqYiKQrySP7MAA1ZrbdzL4FMAfAuHSWJSJpS9LslQB2Zf1cG/3uD5C8l+RqkqvjfMQlIuko+Bt0ZjbDzKrNrDrOZ6Yiko4kzb4bQJ+sn6ui34lIK5Sk2T8GcAHJ/iTbAbgdgH9vBhEpith70JnZCZL3AfgdgDYAZprZptRWJiKpSrS7rJktALAgpbWISAEVdd/4EydO4Ouvv3bl9OzZ011n9erVrvibbrrJXSPOPuuDBw9257zzzjvunCFDhrhztmzZ4or/+OOP3TW8B0EBwMaNG9059fX17px58+a54r0HzgDAlVde6c654YYbXPGnTp3Ku027y4oEQs0uEgg1u0gg1OwigVCziwRCzS4SCDW7SCDU7CKBULOLBELNLhIINbtIINTsIoEo+oEw+/fvd+Wcd9557jodO3Z0xd9zzz3uGsOHD3fn9OvXz51zwQUXuHO8U0QAoEuXLq74F1980V1jwoQJ7pw4k2eeeuopd857773niidzDl1p1p49e9w53oO0mht4okd2kUCo2UUCkeS88X1IvkfyU5KbSD6Q5sJEJF1JXrOfAPCQma0lWQ5gDcnFZvZpSmsTkRTFfmQ3s71mtja6fAjAZuQ4b7yItA6pvGYn2Q/AZQBW5dj23ZCIOLPWRSQdiZudZCcAbwJ40MwONt2ePSTC+5GYiKQn6RTXtsg0+itm9lY6SxKRQkjybjwBvAxgs5n592IQkaJK8sh+JYC/AvDnJD+JvsamtC4RSVmSiTAfAPDvMygiZ0VR940/evSoexjB+++/764zaNAgV/zdd9/trjFgwAB3TocOHdw5ZubO6d27tzvHu7Zjx465a+zbt8+d07ZtW3fOuHHj3DkjR450xe/atevMQU306NHDneM9zuGcc/I/WdfusiKBULOLBELNLhIINbtIINTsIoFQs4sEQs0uEgg1u0gg1OwigVCziwRCzS4SCDW7SCCKeiBMZWUlHn/8cVfO3Llz3XW++uorV3yc02Vdeuml7py6ujp3jvfAIQDYvXu3O6dbt26u+FtuucVdY9OmTe6cOPfNiBEj3DnFOBCouYNU0qpz6tSp/PXd1UXkB0nNLhIINbtIINI4u2wbkv9L8rdpLEhECiONR/YHkBkQISKtWNJTSVcB+AsAL6WzHBEplKSP7P8CYDKAvO/3Z0+EOXDgQMJyIhJXkvPG3whgn5mtaS4ueyKM97NcEUlP0vPG30xyJ4A5yJw/fnYqqxKR1CWZ4vpzM6sys34AbgfwrplNTG1lIpIqfc4uEohU9o03s6UAlqbxt0SkMBhn4kjsYqR5Dwa49dZb3XWGDBniiq+srHTXyMy19Nm6das7Z8KECe6ciooKd87DDz/sil+3bp27Rp8+fdw5VVVV7pyrrrrKneM9eKhv377uGsuWLXPnTJzoe2V877334rPPPsv5j1NP40UCoWYXCYSaXSQQanaRQKjZRQKhZhcJhJpdJBBqdpFAqNlFAqFmFwmEml0kEGp2kUAU9UCY7t2723XXXefKGTt2rLvO6tWrXfEbNmxw1+jUqZM7p76+3p1z6NAhd85DDz3kzjly5Igrvn///u4akydPducMHTrUnRPnQJg9e/a44uNMtxk1apQ7Z//+/a74F154AXv27NGBMCIhU7OLBCLpqaS7knyD5BaSm0n+OK2FiUi6kp6p5mkA75jZbSTbAeiYwppEpABiNzvJLgBGArgLAMzsWwDfprMsEUlbkqfx/QHsB/Dv0ay3l0iWNQ3KHhIRZ6a1iKQjSbOXALgcwHNmdhmARgBTmgZlD4lo3759gnIikkSSZq8FUGtmq6Kf30Cm+UWkFUoyJKIOwC6SA6NfXQPg01RWJSKpS/pu/P0AXoneid8O4CfJlyQihZCo2c3sEwDV6SxFRApJe9CJBCKV8U8t1atXL0yZ8r037Ju1efNmd53a2lpX/IUXXuiu4Z3UAQDz5s1z5wwbNsydU1NT487xHnBx9OhRd426ujp3TmlpqTvnwIED7pwuXbq44jt06OCuEef6p0mP7CKBULOLBELNLhIINbtIINTsIoFQs4sEQs0uEgg1u0gg1OwigVCziwRCzS4SCDW7SCCKeiBMY2MjVq5c6cppaGhw1xk9erQrPs7psmbPnu3OiTNFZt26de6ckhL/3eqdDHTJJZe4a4wZM8ad89FHH7lzli5d6s4ZP368K3758uXuGnGmG1VUVLji58yZk3ebHtlFAqFmFwlE0okw/0ByE8mNJF8j6T/IV0SKInazk6wE8PcAqs3sUgBtANye1sJEJF1Jn8aXACglWYLM6Cff3FsRKZokp5LeDeBXAL4AsBdAg5ktahqXPRHmm2++ib9SEUkkydP4bgDGITMG6jwAZSS/d2K27IkwcT56EpF0JHkafy2AHWa238yOA3gLwBXpLEtE0pak2b8AMJxkR5JEZiKM/1SwIlIUSV6zr0JmvttaABuivzUjpXWJSMqSToR5DMBjKa1FRAqoqPvGl5SUoEePHq6cbdu2uevs2eP7BHDAgAHuGnHebOzYsaM7p1evXu6c7du3u3P69Onjit+xY4e7xmuvvebOmTZtmjunX79+7pxdu3a54quqqtw14gzvmDt3riu+uWEf2l1WJBBqdpFAqNlFAqFmFwmEml0kEGp2kUCo2UUCoWYXCYSaXSQQanaRQKjZRQKhZhcJRFEPhGloaMDChQtdOXGGJDz66KOu+CVLlrhrdOnSxZ3T3EEK+QwePNids2LFCnfO0KFDXfFPPPGEu8Ydd9zhzikrK3PneA+EAoD6+npX/PHjx901Ghsb3TmdO3d2xbdp0ybvNj2yiwRCzS4SCDW7SCDO2OwkZ5LcR3Jj1u+6k1xMclv0vVthlykiSbXkkf0/ADQdizoFwBIzuwDAkuhnEWnFztjsZrYMwNdNfj0OwKzo8iwAt6S7LBFJW9zX7D3NbG90uQ5Az3yB2RNhjh49GrOciCSV+A06MzMA1sz27ybCdOigIa8iZ0vcZq8n2RsAou/70luSiBRC3GafD2BSdHkSgN+ksxwRKZSWfPT2GoAPAQwkWUvypwAeB3AdyW3IzHx7vLDLFJGkzrhvvJnl26H5mpTXIiIFVNQDYTp37oxrr73WlXPy5El3nSlTfB/733TTTe4aDQ0N7pzbbrvNnfPuu++6cyoqKtw5jzzyiCv+scf8U79KS0vdOa+++qo756KLLnLnjBw50hU/atQodw3vwTYAMGbMGFf8qlWr8m7T7rIigVCziwRCzS4SCDW7SCDU7CKBULOLBELNLhIINbtIINTsIoFQs4sEQs0uEgg1u0ggij4RZsGCBa6cOAdPvPnmm65479QNAJg2bZo7Z9myZe6cONf/yy+/dOf06tXLFR9n6kqxDur54osv3DlDhgxxxcc5xVqc++Wcc9J7PNYju0gg1OwigYg7JOKXJLeQXE/ybZJdC7pKEUks7pCIxQAuNbM/BbAVwM9TXpeIpCzWkAgzW2RmJ6IfVwKoKsDaRCRFabxmvxtA3qHrGhIh0jokanaSUwGcAPBKvhgNiRBpHWJ/zk7yLgA3ArgmmgojIq1YrGYnORrAZAB/ZmaH012SiBRC3CER/wagHMBikp+QfL7A6xSRhOIOiXi5AGsRkQLSHnQigSjqgTBlZWUYPny4K2fbtm3uOi+99JIrfuzYse4acQ5Q2bp1qzunb9++7pwePXq4c5599llX/Jw5c9w14hwIUl5e7s7x/hsDgHPPPdcV/+STT7preKfuAHAfONbcpCI9sosEQs0uEgg1u0gg1OwigVCziwRCzS4SCDW7SCDU7CKBULOLBELNLhIINbtIINTsIoEo6oEwJ0+exMGDB105Tz/9tLvOBx984IqfOXOmu8apU6fcOZWVle6cOGpra905d955pyv+mWeecdeIc1DLihUr3DmHD/vPp7Jo0SJXfO/evd014lyXuro6V/zx48fzbtMju0gg1OwigYg1ESZr20MkjaT/AGoRKaq4E2FAsg+A6wH4R2aKSNHFmggT+WdkzjCr00iL/ADEes1OchyA3Wa2rgWx302EaWxsjFNORFLg/uiNZEcAjyLzFP6MzGwGgBkAUFlZqWcBImdJnEf2AQD6A1hHcicyQx3XkuyV5sJEJF3uR3Yz2wDgu1NxRg1fbWb+U4eKSNHEnQgjIj8wcSfCZG/vl9pqRKRgirpvfGlpKS6++GJXTpx9kCdOnOiKj7PP+uzZs905M2bMcOc899xz7pzNmze7c6ZPn+6Kb9eunbvGoEGD3Dlt27Z153Tq1Mmdc8UVV7jiX3/9dXeNOMM7ampqXPHNDVTW7rIigVCziwRCzS4SCDW7SCDU7CKBULOLBELNLhIINbtIINTsIoFQs4sEQs0uEgg1u0gginogzKFDh7B8+XJXzvPPP++us2bNGlc8SXeNqqoqd87ChQvdOffff787Z9WqVe6cjRu/d/LgZjU3jCCfOOsqKytz5+zYscOd0759e1f8sGHD3DW8Ax8AoHv37q74kpL8La1HdpFAqNlFAhF7SATJ+0luIbmJpO9gaBEpulhDIkheDWAcgMFm9iMAv0p/aSKSprhDIv4GwONmdiyK2VeAtYlIiuK+Zr8QwFUkV5F8n+TQfIHZQyKOHDkSs5yIJBW32UsAdAcwHMAjAP6beT6/MrMZZlZtZtWlpaUxy4lIUnGbvRbAW5bxEYBTADTJVaQVi9vs8wBcDQAkLwTQDoCGRIi0Ymfcgy4aEjEKQA+StQAeAzATwMzo47hvAUyy5s5hKyJnXZIhEb6Ts4vIWaU96EQCwWI++ya5H8D/5djUA2f3Nb/qq/4fS/2+ZvYnuTYUtdnzIbnazKpVX/VVv3D0NF4kEGp2kUC0lmb3jzdVfdVXfZdW8ZpdRAqvtTyyi0iBqdlFAlHUZic5muRnJGtITsmxvT3JudH2VST7pVi7D8n3SH4anV3ngRwxo0g2kPwk+vrHtOpHf38nyQ3R316dYztJ/mt0/deTvDzF2gOzrtcnJA+SfLBJTKrXP9dZjkh2J7mY5Lboe7c8uZOimG0kJ6VY/5fRGZbWk3ybZNc8uc3eVwnq/4Lk7qzbeGye3GZ7JRYzK8oXgDYAPgdwPjIHzqwDcEmTmL8F8Hx0+XYAc1Os3xvA5dHlcgBbc9QfBeC3BbwNdgLo0cz2sQAWAiAyhw+vKuB9UYfMDhgFu/4ARgK4HMDGrN9NBzAlujwFwBM58roD2B597xZd7pZS/esBlESXn8hVvyX3VYL6vwDwcAvun2Z7Jc5XMR/ZhwGoMbPtZvYtgDnInNoq2zgAs6LLbwC4Jt9x8l5mttfM1kaXDwHYDKAyjb+donEA/tMyVgLoSrJ3AepcA+BzM8u1N2NqLPdZjrLv41kAbsmRegOAxWb2tZkdALAYTU6NFre+mS0ysxPRjysB+M8JnqB+C7WkV9yK2eyVAHZl/VyL7zfbdzHRHdIAoCLthUQvDy4DkOtE5j8muY7kQpI/Srm0AVhEcg3Je3Nsb8ltlIbbAbyWZ1shrz8A9DSzvdHlOgA9c8QU63a4G5lnUrmc6b5K4r7oZcTMPC9jCnL9g3uDjmQnAG8CeNDMDjbZvBaZp7aDATyDzHH7aRphZpcDGAPg70iOTPnvnxHJdgBuBvB6js2Fvv5/wDLPWc/KZ78kpwI4AeCVPCGFuq+eAzAAwBAAewE8mdLfPaNiNvtuAH2yfq6KfpczhmQJgC4AvkprASTbItPor5jZW023m9lBM/smurwAQFuSqZ2Bx8x2R9/3AXgbmadr2VpyGyU1BsBaM6vPsb6CXv9I/emXJtH3XCcrLejtQPIuADcC+MvoP5zvacF9FYuZ1ZvZSTM7BeDFPH+3INe/mM3+MYALSPaPHl1uBzC/Scx8AKffeb0NwLv57gyv6LX/ywA2m9lTeWJ6nX6PgOQwZG6fVP6zIVlGsvz0ZWTeKGo6c2k+gL+O3pUfDqAh6ylvWu5Anqfwhbz+WbLv40kAfpMj5ncArifZLXqae330u8RIjgYwGcDNZnY4T0xL7qu49bPfg7k1z99tSa/4JX2Hz/nu5Fhk3gX/HMDU6Hf/hMwNDwAdkHl6WQPgIwDnp1h7BDJPGdcD+CT6GgvgZwB+FsXcB2ATMu9+rgRwRYr1z4/+7rqoxunrn12fAH4d3T4bAFSnfPuXIdO8XbJ+V7Drj8x/KnsBHEfmdedPkXkPZgmAbQD+B0D3KLYawEtZuXdH/w5qAPwkxfo1yLwePv1v4PSnP+cBWNDcfZVS/f+K7tv1yDRw76b18/VK0i/tLisSiODeoBMJlZpdJBBqdpFAqNlFAqFmFwmEml0kEGp2kUD8P1Nu1DUTTsvvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import math\n",
    "plt.imshow(norm_hidden_representation.reshape(int(math.sqrt(size_hidden_layer)),int(math.sqrt(size_hidden_layer))), cmap='gray')  # Display the image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "3c98c86b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10,)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## hidden output result \n",
    "\n",
    "ouput_result = np.dot(norm_hidden_representation, weight_hidden_output)\n",
    "ouput_result.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f29aa6c5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Result of the forward pass\n",
    "ouput_result.argmax()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "16af9de2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the function that takes as an input the input layer(vector) and parameters and returns the predicted written number\n",
    "def predict_number(input_layer, weights):\n",
    "    \n",
    "    weights = weight_input_hidden, weight_hidden_output\n",
    "\n",
    "    hidden_representation = np.dot(input_layer, weight_input_hidden)\n",
    "    mi = hidden_representation.min()\n",
    "    ma = hidden_representation.max()\n",
    "\n",
    "    # min max normalization\n",
    "    norm_hidden_representation = (hidden_representation - mi)/(ma-mi)\n",
    "\n",
    "    ouput_result = np.dot(norm_hidden_representation, weight_hidden_output)\n",
    "    # Result of the forward pass\n",
    "    result = ouput_result.argmax()\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "38dc3504",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the function that takes as an input the input layer(vector) and parameters and returns the hidden representation and the output result\n",
    "def forward_pass(input_layer, weights):\n",
    "    \n",
    "    weight_input_hidden, weight_hidden_output = weights\n",
    "    \n",
    "    hidden_representation = np.dot(input_layer, weight_input_hidden)\n",
    "    \n",
    "    # Scaling the hidden layer\n",
    "    mean = np.mean(hidden_representation)\n",
    "    variance = np.var(hidden_representation)\n",
    "    epsilon = 10 **(-5)\n",
    "    scaled_hidden_representation = (hidden_representation - mean)/np.sqrt(variance + epsilon)\n",
    "    \n",
    "    # Calculating the output layer\n",
    "    output_result = np.dot(scaled_hidden_representation, weight_hidden_output)\n",
    "    \n",
    "    return scaled_hidden_representation, output_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "id": "4d037c3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_pass_dropout(input_layer, weights, p=0.5, training=True):\n",
    "    weight_input_hidden, weight_hidden_output = weights\n",
    "    \n",
    "    # Flatten the input_layer\n",
    "    input_layer = input_layer.flatten()\n",
    "    \n",
    "    # Compute the hidden representation\n",
    "    hidden_representation = np.dot(input_layer, weight_input_hidden)\n",
    "    \n",
    "    # Apply dropout to the hidden representation during training\n",
    "    if training:\n",
    "        mask = (np.random.rand(*hidden_representation.shape) > p).astype(np.float32)\n",
    "        hidden_representation = hidden_representation * mask / (1-p)\n",
    "    else:\n",
    "        mask = None\n",
    "    \n",
    "    # Normalize the hidden representation\n",
    "    mean = np.mean(hidden_representation)\n",
    "    variance = np.var(hidden_representation)\n",
    "    epsilon = 10 **(-5)\n",
    "    scaled_hidden_representation = (hidden_representation - mean)/np.sqrt(variance + epsilon)\n",
    "\n",
    "    output_result = np.dot(scaled_hidden_representation, weight_hidden_output)\n",
    "    \n",
    "    return scaled_hidden_representation, output_result, mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1e342005",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAAD3CAYAAADmIkO7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAHOklEQVR4nO3dT4hW1x3H4XPUZgwRR8lCkUAlhEI3IQR12wi6aCEWBpcaDQjixpWQgGYh2blwI1gCEyYI7aJQEDdKCFkKgSAiNBtpqBSDfxISFB1E4+2q0ID3jJl3xvf73nme5fy47z0MfDzi8d63dl1XgDyrxr0A4NnECaHECaHECaHECaHECaHECaHEOQC11qla66e11hu11vu11qu11j+Oe12MRpzDsKaU8p9Syh9KKdOllBOllL/XWreOc1GMpvofQsNUa71WSjnZdd0/xr0WFsfOOUC11k2llN+VUv457rWweHbOgam1/qaUcrGU8q+u6w6Pez0snjgHpNa6qpTyt1LK+lLKn7uuezzmJTGCNeNeAEuj1lpLKZ+WUjaVUv4kzMknzuH4Synl96WUXV3XzY97MYzOX2sHoNb621LKv0spj0opT/5vdLjrur+OZVGMTJwQylEKhBInhBInhBInhGoepdRa/WsRLLOu6+qzfm7nhFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDNrwDk2dasaf/aNm/e3JwfPny4d/baa681rz148GBzPqrZ2dne2fHjx5vX3rlzZ6mXs6LZOSGUOCGUOCGUOCGUOCGUOCGUOCFU7bquf1hr/3AFO3fuXHO+b9++F7SSF2tubq45/+ijj5rz7777bimXMxhd19Vn/dzOCaHECaHECaHECaHECaHECaHECaE8z8lze//995vzhZ5zPXDgwFIuZ/DsnBBKnBBKnBBKnBBKnBBKnBDKUcoiXLx4sTlf6JGxx48f986+//77Ra3pea1du7Y537hx46I/++WXX27OV69e3Zz//PPPi773ENk5IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZRXYy7C9PR0c753797m/IcffuidnT9/fjFLem47d+5szltnuC+99NJI937rrbea82vXro30+ZPKqzFhwogTQokTQokTQokTQokTQokTQjnn5BeuXLnSO1vonHIhp06das4//PDDkT5/UjnnhAkjTgglTgglTgglTgglTgglTgjlvbUrzNtvv92cb926ddnuffbs2WX77CGyc0IocUIocUIocUIocUIocUIocUIo55wrzNGjR5vzDRs2vJiFsCA7J4QSJ4QSJ4QSJ4QSJ4QSJ4RylDIw7777bnO+Z8+eZbv3l19+2ZzfvXt32e49RHZOCCVOCCVOCCVOCCVOCCVOCCVOCOWcc8Is9EjXxx9/PNL1o7h8+XJzPj8/v2z3HiI7J4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4RyzhlmoXPIubm55vzNN99cwtX80u3bt5vzTz75ZNnuvRLZOSGUOCGUOCGUOCGUOCGUOCGUOCGUc84x2LZtW+/sxIkTzWuX872zpZRy69at3tnMzEzz2ps3by71clY0OyeEEieEEieEEieEEieEEieEEieEcs65COvWrWvOd+/e3Zx/8MEHvbMdO3Ysak1L5caNG72zb7755gWuBDsnhBInhBInhBInhBInhBInhKpd1/UPa+0fTrB33nmnOT9z5kxzPjU11Zy/8cYbv3ZJE+HChQvN+f79+5vz+/fvL+VyBqPruvqsn9s5IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IdRgzzk3btzYO/v222+b105PTy/1claE1uNmpZRy5MiR5vzSpUtLuZyJ4ZwTJow4IZQ4IZQ4IZQ4IZQ4IZQ4IdRgzzl37drVO/v8889f4Er4nwcPHjTnr7/+eu/s7t27S72cGM45YcKIE0KJE0KJE0KJE0KJE0KJE0IN9isAr1+/Pu4lLIubN2825wu9W3ZmZqY537Rp069e0/N65ZVXmvNjx471zlpfmzhUdk4IJU4IJU4IJU4IJU4IJU4INdhHxl599dXe2aiPH/3000/N+Weffdac37p1q3c2OzvbvPbJkyfN+b1795rz9evXN+dr1vSfrp0+fbp57XvvvdecL+TRo0e9sy1btjSv/fHHH0e69zh5ZAwmjDghlDghlDghlDghlDghlDgh1GAfGZufn++dXb16tXlt6xyylFJOnjzZnH/11VfN+TgtdA7acujQoeZ81ar2n/X79u1rzqempnpntT7zKHDQ7JwQSpwQSpwQSpwQSpwQSpwQSpwQarDnnA8fPuydbd++vXlt6xnXUkp5+vTpotY06RZ6lvSLL75ozhc65/z66697Z61z66Gyc0IocUIocUIocUIocUIocUIocUKowb63FiaF99bChBEnhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhGp+BSAwPnZOCCVOCCVOCCVOCCVOCCVOCPVfGWM+6h2PFHIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(289,)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Test\n",
    "visualize_image(image, label)\n",
    "forward_pass(input_layer, weights)[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c65cdad",
   "metadata": {},
   "source": [
    "## 4- Implementing the backward pass:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "613b62f0",
   "metadata": {},
   "source": [
    "This section would modify the *input_hidden_weights* and *hidden_output_weights* in order to make better predictions.\n",
    "\n",
    "The weights modifications would be based on the backpropagation algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "78149688",
   "metadata": {},
   "outputs": [],
   "source": [
    "def MSE(predicted, desired):\n",
    "    # This function takes as input numpy arrays and not lists\n",
    "    diff = 1/2 * ((predicted - desired)**2).sum()\n",
    "    return diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "6e7365aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "def full_backward_pass(weights, image, label, lr = 0.01, lamda = 0.01):\n",
    "    # We only modify the weights between the last and the penultimate layer\n",
    "    # weights is a tuple of matrices : weight_input_hidden & weight_hidden_output\n",
    "\n",
    "    size_hidden_layer = SIZE_HIDDEN_LAYER\n",
    "    size_output_layer = SIZE_OUTPUT_LAYER\n",
    "\n",
    "    # one hot encoding the label\n",
    "    desired = np.zeros(size_output_layer)\n",
    "    desired[int(label)] = 1\n",
    "\n",
    "    weight_input_hidden, weight_hidden_output = weights\n",
    "    new_weight_input_hidden, new_weight_hidden_output = weight_input_hidden.copy(), weight_hidden_output.copy()\n",
    "\n",
    "    input_layer = image.flatten() # flattening the image\n",
    "\n",
    "    hidden_representation, output_result = forward_pass(input_layer, (weight_input_hidden, weight_hidden_output)) # forward pass\n",
    "\n",
    "    output_result =  tf.nn.softmax(output_result) # softmax operation\n",
    "\n",
    "    delta_matrix = np.zeros((size_hidden_layer, size_output_layer))\n",
    "\n",
    "    # gradient descent iteration # backward pass for the second layer\n",
    "    for k in range(size_output_layer):\n",
    "        for j in range(size_hidden_layer):\n",
    "            delta = (output_result[k] - desired[k]) * hidden_representation[j] * (1 - output_result[k]) * output_result[k]\n",
    "            new_weight_hidden_output[j][k] = (1-lamda) * new_weight_hidden_output[j][k] - lr*delta\n",
    "\n",
    "            delta_matrix[j][k] = delta\n",
    "\n",
    "    for j in range(size_hidden_layer):\n",
    "        for i in range(size_input_layer):\n",
    "            delta = np.sum(new_weight_hidden_output[j]* (delta_matrix[j]) * input_layer[i] / hidden_representation[j])\n",
    "            new_weight_input_hidden[i][j] = (1-lamda) * new_weight_input_hidden[i][j] - lr*delta\n",
    "\n",
    "\n",
    "    return new_weight_input_hidden, new_weight_hidden_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5461fd66",
   "metadata": {},
   "outputs": [],
   "source": [
    "def full_backward_pass_cross_entropy(weights, image, label, lr=0.01, lamda=0.01):\n",
    "    size_hidden_layer = SIZE_HIDDEN_LAYER\n",
    "    size_output_layer = SIZE_OUTPUT_LAYER\n",
    "\n",
    "    # one hot encoding the label\n",
    "    desired = np.zeros(size_output_layer)\n",
    "    desired[int(label)] = 1\n",
    "\n",
    "    weight_input_hidden, weight_hidden_output = weights\n",
    "    new_weight_input_hidden, new_weight_hidden_output = weight_input_hidden.copy(), weight_hidden_output.copy()\n",
    "\n",
    "    input_layer = image.flatten()  # flattening the image\n",
    "\n",
    "    hidden_representation, logits = forward_pass(input_layer, (weight_input_hidden, weight_hidden_output))  # forward pass\n",
    "    output_result = tf.nn.softmax(logits)  # softmax operation\n",
    "\n",
    "    # Gradient of cross-entropy loss w.r.t. logits\n",
    "    d_logits = output_result - desired\n",
    "\n",
    "    # Backward pass for the second layer\n",
    "    d_weight_hidden_output = np.outer(hidden_representation, d_logits)\n",
    "    new_weight_hidden_output = (1 - lamda) * weight_hidden_output - lr * d_weight_hidden_output\n",
    "\n",
    "    # Backward pass for the first layer\n",
    "    d_hidden_representation = np.dot(d_logits, weight_hidden_output.T)\n",
    "    d_weight_input_hidden = np.outer(input_layer, d_hidden_representation)\n",
    "    new_weight_input_hidden = (1 - lamda) * weight_input_hidden - lr * d_weight_input_hidden\n",
    "\n",
    "    return new_weight_input_hidden, new_weight_hidden_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "9b2992cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def full_backward_pass_cross_entropy_adam(weights, image, label, t, m_prev_1, m_prev_2, v_prev_1, v_prev_2, lr=0.001, lamda=0.01, beta1=0.9, beta2=0.999, epsilon=1e-8):\n",
    "    size_hidden_layer = 300\n",
    "    size_output_layer = 10\n",
    "\n",
    "    # one hot encoding the label\n",
    "    desired = np.zeros(size_output_layer)\n",
    "    desired[int(label)] = 1\n",
    "\n",
    "    weight_input_hidden, weight_hidden_output = weights\n",
    "\n",
    "    input_layer = image.flatten()  # flattening the image\n",
    "    hidden_representation, logits = forward_pass(input_layer, (weight_input_hidden, weight_hidden_output))  # forward pass\n",
    "    output_result = tf.nn.softmax(logits)  # softmax operation\n",
    "\n",
    "    # Gradient of cross-entropy loss w.r.t. logits\n",
    "    d_logits = output_result - desired\n",
    "\n",
    "    # Backward pass for the second layer\n",
    "    d_weight_hidden_output = np.outer(hidden_representation, d_logits)\n",
    "    # Adam for weight_hidden_output\n",
    "    m_t_2 = beta1 * m_prev_2 + (1 - beta1) * d_weight_hidden_output\n",
    "    v_t_2 = beta2 * v_prev_2 + (1 - beta2) * d_weight_hidden_output**2\n",
    "    m_t_hat_2 = m_t_2 / (1 - beta1**t)\n",
    "    v_t_hat_2 = v_t_2 / (1 - beta2**t)\n",
    "    new_weight_hidden_output = weight_hidden_output - lr * m_t_hat_2 / (np.sqrt(v_t_hat_2) + epsilon)\n",
    "\n",
    "    # Backward pass for the first layer\n",
    "    d_hidden_representation = np.dot(d_logits, weight_hidden_output.T)\n",
    "    d_weight_input_hidden = np.outer(input_layer, d_hidden_representation)\n",
    "    # Adam for weight_input_hidden\n",
    "    m_t_1 = beta1 * m_prev_1 + (1 - beta1) * d_weight_input_hidden\n",
    "    v_t_1 = beta2 * v_prev_1 + (1 - beta2) * d_weight_input_hidden**2\n",
    "    m_t_hat_1 = m_t_1 / (1 - beta1**t)\n",
    "    v_t_hat_1 = v_t_1 / (1 - beta2**t)\n",
    "    new_weight_input_hidden = weight_input_hidden - lr * m_t_hat_1 / (np.sqrt(v_t_hat_1) + epsilon)\n",
    "\n",
    "    return new_weight_input_hidden, new_weight_hidden_output, m_t_1, m_t_2, v_t_1, v_t_2\n",
    "\n",
    "# Note: This function now also returns the updated moments m and v for both layers, which should be passed in the next call for updates.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "e1b6f4e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def full_backward_pass_cross_entropy_adam_dropout(weights, image, label, mask, t, m_prev_1, m_prev_2, v_prev_1, v_prev_2, lr=0.001, lamda=0.01, beta1=0.9, beta2=0.999, epsilon=1e-8):\n",
    "   \n",
    "\n",
    "    desired = np.zeros(size_output_layer)\n",
    "    desired[int(label)] = 1\n",
    "\n",
    "    weight_input_hidden, weight_hidden_output = weights\n",
    "    input_layer = image.flatten()\n",
    "    hidden_representation, logits, _ = forward_pass_dropout(input_layer, (weight_input_hidden, weight_hidden_output))\n",
    "    output_result = tf.nn.softmax(logits)\n",
    "\n",
    "    d_logits = output_result - desired\n",
    "    d_weight_hidden_output = np.outer(hidden_representation, d_logits)\n",
    "\n",
    "    m_t_2 = beta1 * m_prev_2 + (1 - beta1) * d_weight_hidden_output\n",
    "    v_t_2 = beta2 * v_prev_2 + (1 - beta2) * d_weight_hidden_output**2\n",
    "    m_t_hat_2 = m_t_2 / (1 - beta1**t)\n",
    "    v_t_hat_2 = v_t_2 / (1 - beta2**t)\n",
    "    new_weight_hidden_output = weight_hidden_output - lr * m_t_hat_2 / (np.sqrt(v_t_hat_2) + epsilon)\n",
    "\n",
    "    d_hidden_representation = np.dot(d_logits, weight_hidden_output.T) * mask\n",
    "    d_weight_input_hidden = np.outer(input_layer, d_hidden_representation)\n",
    "\n",
    "    m_t_1 = beta1 * m_prev_1 + (1 - beta1) * d_weight_input_hidden\n",
    "    v_t_1 = beta2 * v_prev_1 + (1 - beta2) * d_weight_input_hidden**2\n",
    "    m_t_hat_1 = m_t_1 / (1 - beta1**t)\n",
    "    v_t_hat_1 = v_t_1 / (1 - beta2**t)\n",
    "    new_weight_input_hidden = weight_input_hidden - lr * m_t_hat_1 / (np.sqrt(v_t_hat_1) + epsilon)\n",
    "\n",
    "    return new_weight_input_hidden, new_weight_hidden_output, m_t_1, m_t_2, v_t_1, v_t_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "16af90cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# testing the backward pass \n",
    "\n",
    "# preparing the args\n",
    "random_idx += 1 \n",
    "\n",
    "image = x_train[random_idx]; input_layer = image.flatten()\n",
    "label = y_train[random_idx]\n",
    "\n",
    "# one hot encoding\n",
    "desired = np.zeros(size_output_layer)\n",
    "desired[int(label)] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "2a603f18",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAAD3CAYAAADmIkO7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAHVUlEQVR4nO3dT4hV5wHG4e9rYghhEBTqRgkS/w1UKCSCloQwoFBId8ZFuxAXEaTiQsSAKxfuioogSUlwk0CDLsy2s4hlClYIYiGLKI0MaEENJKJYzOAg5WTRPyDM/W684zjvvfM8kEV8OTMH4Zdj8uXMrV3XFSDPzxb7BoC5iRNCiRNCiRNCiRNCiRNCiRNCiXNE1Fr/VGv9ttb6r1rr9Vrr3sW+J+an+p8QRkOt9RellOmu62ZrreOllL+WUn7Tdd3fF/fOGJQn54jouu5q13Wz//vb//61bhFviXkS5wiptf6x1jpTSvlHKeXbUsqfF/mWmAd/rB0xtdYXSim/KqVMlFL+0HXd48W9IwblyTliuq77d9d1fyulrCml/H6x74fBiXN0vVj8O+dQE+cIqLWuqrX+ttY6Vmt9odb661LK70opf1nse2Nw/p1zBNRaf15KOV9K+WX5zz9w/1lKOd113ZlFvTHmRZwQyh9rIZQ4IZQ4IZQ4IdSLrbHW6r8WwQLruq7O9euenBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBBKnBCq+RGADGZsbKy5v/baaz23o0ePNq999913m3vXtT+18cSJE839wYMHA187Ozvb3Hk6npwQSpwQSpwQSpwQSpwQSpwQSpwQqrbOxWqt7UMz5vTJJ5809927dz+fG3nGpqammvvk5GRzP3ny5LO8nZHRdV2d69c9OSGUOCGUOCGUOCGUOCGUOCGUOCGUc84B9Htfs9954Ouvv95zu3btWvPa06dPN/e7d+8294mJieZ+4MCB5t7y/fffN/e33nqruU9PTw/8vYeZc04YMuKEUOKEUOKEUOKEUOKEUI5SFsDhw4eb+8aNG3tu77//fvPa1o+u/Cn6HQPN5+vfu3evub/55pvN/fr16wN/72HmKAWGjDghlDghlDghlDghlDghlDghlI8AXAD9PipvIY2Pjzf3s2fPLtj3PnfuXHNfqueYg/LkhFDihFDihFDihFDihFDihFDihFDe5xwx33zzTXNfv359c3/48GHPrd/57fHjx5v7o0ePmvtS5X1OGDLihFDihFDihFDihFDihFDihFDe5xwyV65cae7r1q1r7v3OGt95552e26VLl5rX8mx5ckIocUIocUIocUIocUIocUIocUIo55xhtm7d2tzXrl3b3Gud89XA/9u+fXtz//LLL5s7z48nJ4QSJ4QSJ4QSJ4QSJ4QSJ4RylBKm34+fXLFixby+/t69e5v73bt3e27T09Pz+t48HU9OCCVOCCVOCCVOCCVOCCVOCCVOCOUjAMPs2bOnuX/wwQfN/ZVXXpnX95+Zmem5HTp0qHnt+fPnm/v9+/cHuqdR5yMAYciIE0KJE0KJE0KJE0KJE0KJE0J5nzPMp59+2txb59KllLJz587mPj4+3tw3bNjQc/voo4+a17700kvN/cMPP2zuPMmTE0KJE0KJE0KJE0KJE0KJE0KJE0J5n3OJWb16dXP/4osvem6bNm1qXjs1NdXcd+zY0dyXKu9zwpARJ4QSJ4QSJ4QSJ4QSJ4QSJ4RyzskTWu97Xr16tXnt48ePm/u2bdua+1dffdXcR5VzThgy4oRQ4oRQ4oRQ4oRQ4oRQfjQmT2h9BGA/t27dau7ffffdwF97KfLkhFDihFDihFDihFDihFDihFDihFDOOXnC8uXLB7725s2bzf3OnTsDf+2lyJMTQokTQokTQokTQokTQokTQokTQjnnXGJefvnl5v75558/pzuhH09OCCVOCCVOCCVOCCVOCCVOCCVOCLUkzznXrFnT3Dds2NDcx8bGmvv+/fub++3bt3tun332WfPay5cvN/cffvihuR85cqS5r1+/vuf28OHD5rUff/xxc+fpeHJCKHFCKHFCKHFCKHFCKHFCqNp1Xe+x1t7jAlu5cmVzf++995r7rl27em6rVq1qXvvqq68298X09ddfN/dHjx419zfeeKO511p7bocPH25ee+rUqebO3Lqum/M33ZMTQokTQokTQokTQokTQokTQokTQsW+Mtbvo+gmJiaa+5YtW3pu165da1575syZ5t7P5ORkc3/77bd7bgcPHmxeu3nz5kFu6Se7dOlSz+3ixYsL+r15kicnhBInhBInhBInhBInhBInhBInhIp9n7OfZcuWNffWj6+cnZ1tXjszMzPQPf1Ua9eu7bnt27eveW2/91hv3LjR3I8dO9bcL1y40HPr9/vGYLzPCUNGnBBKnBBKnBBKnBBKnBBKnBBqaM85YVQ454QhI04IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4IJU4I1fwIQGDxeHJCKHFCKHFCKHFCKHFCKHFCqB8BEv9muTzII7sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_image(image, label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "136f29b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# skip if you don't want to perform the backward pass\n",
    "#new_weight_input_hidden, new_weight_hidden_output = full_backward_pass(weights, image, label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "c0ab06bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.25977721860368896"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#_, new_output = forward_pass(input_layer, (new_weight_input_hidden, new_weight_hidden_output))\n",
    "# norm_new_output = np.array(tf.nn.softmax(new_output)) # transforming the logits to probs\n",
    "# MSE(norm_new_output, desired)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "17557509",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9955993105371983"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#_, old_output = forward_pass(input_layer, (weight_input_hidden, weight_hidden_output))\n",
    "# norm_old_output = np.array(tf.nn.softmax(old_output))\n",
    "# MSE(norm_old_output, desired)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "cd8066d4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#np.argmax(new_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f8ed5ae",
   "metadata": {},
   "source": [
    "## 5 - Training and Validation loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "51d41464",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "def train(weights, train_data, val_data=None):\n",
    "    weight_input_hidden, weight_hidden_output = weights\n",
    "    images, labels = train_data\n",
    "    \n",
    "    # List to store weights periodically\n",
    "    saved_weights = []\n",
    "\n",
    "    for idx, (image, label) in tqdm(enumerate(zip(images, labels))):\n",
    "        #new_weight_input_hidden, new_weight_hidden_output = full_backward_pass_cross_entropy(weights, image, label)\n",
    "        new_weight_input_hidden, new_weight_hidden_output = full_backward_pass(weights, image, label)\n",
    "        weights = new_weight_input_hidden, new_weight_hidden_output\n",
    "        \n",
    "        # Save weights every N iterations (e.g., every 10 images)\n",
    "        #if idx % 10 == 0:\n",
    "            #saved_weights.append(new_weight_input_hidden.copy())  \n",
    "        \n",
    "        if val_data and (idx % 3 == 0) and (idx != 0):\n",
    "            metric_value = validate(weights, val_data)\n",
    "            print(f\"Metric value: {idx}th image (number: {label}): {metric_value}\")\n",
    "\n",
    "    return weights, saved_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "id": "20a3e6cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_adam(weights, train_data, val_data=None, lr=0.001, lamda=0.01, beta1=0.9, beta2=0.999, epsilon=1e-8):\n",
    "    weight_input_hidden, weight_hidden_output = weights\n",
    "    images, labels = train_data\n",
    "    \n",
    "    # List to store weights periodically\n",
    "    saved_weights = []\n",
    "\n",
    "    # Initialize moments for Adam optimizer\n",
    "    m_prev_1 = np.zeros_like(weight_input_hidden)\n",
    "    m_prev_2 = np.zeros_like(weight_hidden_output)\n",
    "    v_prev_1 = np.zeros_like(weight_input_hidden)\n",
    "    v_prev_2 = np.zeros_like(weight_hidden_output)\n",
    "    \n",
    "    for idx, (image, label) in tqdm(enumerate(zip(images, labels))):\n",
    "        new_weight_input_hidden, new_weight_hidden_output, m_prev_1, m_prev_2, v_prev_1, v_prev_2 = full_backward_pass_cross_entropy_adam(\n",
    "            weights, image, label, idx+1, m_prev_1, m_prev_2, v_prev_1, v_prev_2, lr, lamda, beta1, beta2, epsilon\n",
    "        )\n",
    "        weights = new_weight_input_hidden, new_weight_hidden_output\n",
    "        \n",
    "        # Save weights every N iterations (e.g., every 10 images)\n",
    "        if idx % 10 == 0:\n",
    "            saved_weights.append(new_weight_input_hidden.copy())  \n",
    "        \n",
    "        if val_data and (idx % 10 == 0) and (idx != 0):\n",
    "            metric_value = validate(weights, val_data)\n",
    "            print(f\"Metric value: {idx}th image (number: {label}): {metric_value}\")\n",
    "\n",
    "    return weights, saved_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "id": "2987b82b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_dropout(weights, train_data, val_data=None):\n",
    "    weight_input_hidden, weight_hidden_output = weights\n",
    "    images, labels = train_data\n",
    "    saved_weights = []\n",
    "\n",
    "    # Initialize moments for Adam optimizer\n",
    "    m_prev_1 = np.zeros_like(weight_input_hidden)\n",
    "    m_prev_2 = np.zeros_like(weight_hidden_output)\n",
    "    v_prev_1 = np.zeros_like(weight_input_hidden)\n",
    "    v_prev_2 = np.zeros_like(weight_hidden_output)\n",
    "\n",
    "    for idx, (image, label) in tqdm(enumerate(zip(images, labels))):\n",
    "        _, _, mask = forward_pass_dropout(image, weights)\n",
    "        new_weight_input_hidden, new_weight_hidden_output, m_prev_1, m_prev_2, v_prev_1, v_prev_2 = full_backward_pass_cross_entropy_adam_dropout(\n",
    "            weights, image, label, mask, idx+1, m_prev_1, m_prev_2, v_prev_1, v_prev_2\n",
    "        )\n",
    "        weights = new_weight_input_hidden, new_weight_hidden_output\n",
    "\n",
    "        if idx % 10 == 0:\n",
    "            saved_weights.append(new_weight_input_hidden.copy())\n",
    "\n",
    "        if val_data and (idx % 20 == 0) and (idx != 0):\n",
    "            metric_value = validate_dropout(weights, val_data)\n",
    "            print(f\"Metric value: {idx}th image (number: {label}): {metric_value}\")\n",
    "\n",
    "    return weights, saved_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "d5d8b373",
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate(weights, val_data):\n",
    "    weight_input_hidden, weight_hidden_output = weights\n",
    "    images, labels = val_data\n",
    "    \n",
    "    predicted_labels = []\n",
    "    for image in images:\n",
    "        input_layer = image.flatten()\n",
    "        _, logits = forward_pass(input_layer, weights)\n",
    "        predicted_label = np.argmax(logits)\n",
    "        predicted_labels.append(predicted_label)\n",
    "        \n",
    "    return sum(predicted_labels == labels)/labels.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "f5bf4d15",
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate_dropout(weights, val_data):\n",
    "    weight_input_hidden, weight_hidden_output = weights\n",
    "    images, labels = val_data\n",
    "    \n",
    "    predicted_labels = []\n",
    "    for image in images:\n",
    "        _, logits, _ = forward_pass_dropout(image, weights, training=False)\n",
    "        predicted_label = np.argmax(logits)\n",
    "        predicted_labels.append(predicted_label)\n",
    "        \n",
    "    return sum(predicted_labels == labels)/labels.shape[0]\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "f58e81d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preparing data\n",
    "train_size = 6000\n",
    "val_size = 2000\n",
    "\n",
    "train_data = x_train[1000 :1000 +train_size], y_train[1000: 1000 + train_size]\n",
    "val_data = x_test[:val_size], y_test[:val_size]\n",
    "# weights_init_xavier = init_weights_xavier()\n",
    "# weights_init_he = init_weights_he()\n",
    "# weights_init = init_weights()\n",
    "# weights = weights_init_he"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "a8457c52",
   "metadata": {},
   "outputs": [],
   "source": [
    "val_size = 2000\n",
    "val_data = x_test[:val_size], y_test[:val_size]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "id": "e9100f04",
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = weights_init_he"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9595fe5f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "4it [01:15, 19.30s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metric value: 3th image (number: 0): 0.096\n"
     ]
    }
   ],
   "source": [
    "weights, saved_weights = train(weights, train_data, val_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "id": "fb39d3cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8155"
      ]
     },
     "execution_count": 196,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validate_dropout(weights,val_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "id": "0e61bdf9",
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "too many values to unpack (expected 2)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[183], line 3\u001b[0m\n\u001b[0;32m      1\u001b[0m c \u001b[39m=\u001b[39m \u001b[39m0\u001b[39m\n\u001b[0;32m      2\u001b[0m \u001b[39mfor\u001b[39;00m weightt \u001b[39min\u001b[39;00m saved_weights :\n\u001b[1;32m----> 3\u001b[0m     \u001b[39mif\u001b[39;00m validate(weightt, val_data)\u001b[39m>\u001b[39mc :\n\u001b[0;32m      4\u001b[0m         c \u001b[39m=\u001b[39m validate(weightt, val_data)\n\u001b[0;32m      5\u001b[0m \u001b[39mprint\u001b[39m(c)\n",
      "Cell \u001b[1;32mIn[177], line 2\u001b[0m, in \u001b[0;36mvalidate\u001b[1;34m(weights, val_data)\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mvalidate\u001b[39m(weights, val_data):\n\u001b[1;32m----> 2\u001b[0m     weight_input_hidden, weight_hidden_output \u001b[39m=\u001b[39m weights\n\u001b[0;32m      3\u001b[0m     images, labels \u001b[39m=\u001b[39m val_data\n\u001b[0;32m      5\u001b[0m     predicted_labels \u001b[39m=\u001b[39m []\n",
      "\u001b[1;31mValueError\u001b[0m: too many values to unpack (expected 2)"
     ]
    }
   ],
   "source": [
    "c = 0\n",
    "for weightt in saved_weights :\n",
    "    if validate(weightt, val_data)>c :\n",
    "        c = validate(weightt, val_data)\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "id": "4f9e2e81",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "def visualize_and_save_weights(saved_weights, image_shape, save_path=\"./\"):\n",
    "    for idx, weight_input_hidden in enumerate(saved_weights):\n",
    "        plt.figure(figsize=(10, 6))\n",
    "        num_filters = weight_input_hidden.shape[1]\n",
    "\n",
    "        for i in range(num_filters):\n",
    "            plt.subplot(num_filters // 10 + 1, 10, i + 1)  # Display 10 filters per row\n",
    "            filter_img = weight_input_hidden[:, i].reshape(image_shape)\n",
    "            plt.imshow(filter_img, cmap='gray')\n",
    "            plt.axis('off')\n",
    "        \n",
    "        # Save the current figure as an image\n",
    "        plt.savefig(f\"{save_path}/weights_{idx}.png\")\n",
    "        plt.close()\n",
    "\n",
    "\n",
    "# Assuming the images are 28x28 (like in MNIST)\n",
    "image_shape = (28, 28)\n",
    "save_path = \"./\"\n",
    "saved_weights_tst = saved_weights[-11:]\n",
    "visualize_and_save_weights(saved_weights_tst, image_shape, save_path=\"./\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "id": "13bbcdd4",
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'plotly'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[185], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[39mimport\u001b[39;00m \u001b[39mplotly\u001b[39;00m\u001b[39m.\u001b[39;00m\u001b[39mexpress\u001b[39;00m \u001b[39mas\u001b[39;00m \u001b[39mpx\u001b[39;00m\n\u001b[0;32m      3\u001b[0m fig \u001b[39m=\u001b[39m px\u001b[39m.\u001b[39mhistogram(train_data[\u001b[39m1\u001b[39m], nbins\u001b[39m=\u001b[39m\u001b[39m10\u001b[39m, title\u001b[39m=\u001b[39m\u001b[39m'\u001b[39m\u001b[39mTraining data distribution\u001b[39m\u001b[39m'\u001b[39m)\n\u001b[0;32m      4\u001b[0m fig\u001b[39m.\u001b[39mupdate_xaxes(tickvals\u001b[39m=\u001b[39mnp\u001b[39m.\u001b[39marange(\u001b[39m0\u001b[39m, \u001b[39m10\u001b[39m))\n",
      "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'plotly'"
     ]
    }
   ],
   "source": [
    "import plotly.express as px\n",
    "\n",
    "fig = px.histogram(train_data[1], nbins=10, title='Training data distribution')\n",
    "fig.update_xaxes(tickvals=np.arange(0, 10))\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d6b8f8d",
   "metadata": {},
   "source": [
    "## 6- Inference:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "id": "ee1fb8ff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGbCAYAAAAr/4yjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAKF0lEQVR4nO3cMWiVVwPG8XM1CIouoggGl6JUbCeVOhR10Q5FHaRiXSrS2oLYoTioi+IgDuJmKUJFFMRSEEEHwaFDB0G6CFZJRQehiFiqiUUjBrnf1Odr/b7hnldzX5P8fmO5D+9Bmvx9jZ5Ot9vtFgAopUxr+wAAvD1EAYAQBQBCFAAIUQAgRAGAEAUAQhQACFEAIEQBgBAFpqybN2+WLVu2lHfeeafMmjWrzJs3r6xZs6ZcunSp7aNBawbaPgC05d69e+Wvv/4q27dvLwsXLizPnj0r58+fL5s2bSonTpwoX375ZdtHhL7ruBAP/uvly5dlxYoV5fnz52VoaKjt40Df+eMj+Ifp06eXRYsWleHh4baPAq3wx0dMeU+fPi2jo6NlZGSkXLx4sVy+fLls3bq17WNBK0SBKW/Pnj3lxIkTpZRSpk2bVjZv3lyOHz/e8qmgHX6mwJQ3NDRUfv/993L//v3y448/lhkzZpTvvvuuLFiwoO2jQd+JArzio48+KsPDw+XatWul0+m0fRzoKz9ohld88skn5Zdffim3b99u+yjQd6IArxgdHS2llDIyMtLySaD/RIEp6+HDh//z38bGxsqZM2fKzJkzy7Jly1o4FbTL3z5iyvrqq6/KkydPypo1a8rg4GB58OBBOXv2bBkaGirHjh0rs2fPbvuI0Hd+0MyU9cMPP5STJ0+WGzdulD///LPMmTOnrFixonz99ddl06ZNbR8PWiEKAISfKQAQogBAiAIAIQoAhCgAEKIAQPT8j9dcDAYwsfXyLxC8KQAQogBAiAIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIAMdD2AZiYFi9eXL355ptvGj1ryZIl1ZuzZ89Wb06fPl29gcnGmwIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAdLrdbrenD3Y6430WJpDvv/++erNjx45xOMn/98cff1RvPvzww+rN3bt3qzfQll6+3XtTACBEAYAQBQBCFAAIUQAgRAGAEAUAQhQACFEAIEQBgBAFAEIUAIiBtg/AxHTlypXqzaxZsxo9a+vWrdWb+fPnV2/27dtXvdm5c2f1Bt5m3hQACFEAIEQBgBAFAEIUAAhRACBEAYAQBQBCFAAIUQAgRAGAEAUAotPtdrs9fbDTGe+zMMlNm9bs9yB79uyp3hw+fLh6MzY2Vr0ZHBys3gwPD1dv4E3o5du9NwUAQhQACFEAIEQBgBAFAEIUAAhRACBEAYAQBQBCFAAIUQAgRAGAcCEek9KpU6eqN5999ln15urVq9WbDRs2VG9KKWVkZKTRDv7mQjwAqogCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQLgllUlp6dKl1Ztbt25Vb5p8XTQ5Wyml/Pbbb4128De3pAJQRRQACFEAIEQBgBAFAEIUAAhRACBEAYAQBQBCFAAIUQAgRAGAGGj7AEwdu3fvbrTbtWtX9WbGjBnVmx7vhvyXsbGx6s3Lly+rN9Av3hQACFEAIEQBgBAFAEIUAAhRACBEAYAQBQBCFAAIUQAgRAGAEAUAwoV49M2GDRsa7d599903fJI358iRI9WbO3fujMNJ4M3wpgBAiAIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQnW632+3pg53OeJ+FSe6LL75otDtw4ED1ZnBwsNGzaj19+rR6s3z58kbPcpEer6uXb/feFAAIUQAgRAGAEAUAQhQACFEAIEQBgBAFAEIUAAhRACBEAYAQBQBCFAAIt6Ty1ps7d271ZtWqVdWbffv2VW9Wr15dvfnpp5+qN6WUsm7dukY7+JtbUgGoIgoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAuBAPXkOPXz6vvSmllKNHj1Zv9u7d2+hZTE4uxAOgiigAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIA4UI8eA379++v3hw6dKjRs0ZHR6s3GzdurN78/PPP1RsmBhfiAVBFFAAIUQAgRAGAEAUAQhQACFEAIEQBgBAFAEIUAAhRACBEAYAYaPsAMJEdOXKkerN27dpGz1q/fn31ZtWqVdUbF+JNbd4UAAhRACBEAYAQBQBCFAAIUQAgRAGAEAUAQhQACFEAIEQBgBAFAMKFeNBn27Zta7RrclHdypUrGz2LqcubAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgDhllTos8ePHzfa3blzp3rz8ccfV2/ef//96s2vv/5aveHt5E0BgBAFAEIUAAhRACBEAYAQBQBCFAAIUQAgRAGAEAUAQhQACFEAIDrdbrfb0wc7nfE+C5Pcp59+2mh3/fr16s3Q0FCjZ/XDe++912h348aN6k2TX4dly5ZVb5gYevl2700BgBAFAEIUAAhRACBEAYAQBQBCFAAIUQAgRAGAEAUAQhQACFEAIAbaPgAT0/Lly6s3p06davSsFy9eVG9u3bpVvblw4UL1ponPP/+80a7Huyv/5fnz542exdTlTQGAEAUAQhQACFEAIEQBgBAFAEIUAAhRACBEAYAQBQBCFAAIUQAgXIhHI8+ePaveNLnYrpRSZs+eXb354IMP+rLpdDrVmyYX2zV17ty5vj2LycGbAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQIgCACEKAIQoABCiAEB0uj3eztXk4i/4p/Xr1zfanTx5snozODjY6Fm1+nkh3tWrV6s3GzdurN4MDw9Xb5gYevl/z5sCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQIgCAOGWVN56c+fOrd7s3r27enPw4MHqTZOviybPKaWUb7/9tnrz6NGjRs9icnJLKgBVRAGAEAUAQhQACFEAIEQBgBAFAEIUAAhRACBEAYAQBQBCFAAIF+IBTBEuxAOgiigAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQIgCACEKAIQoABCiAECIAgAhCgCEKAAQogBAiAIAIQoAhCgAEKIAQIgCACEKAMRArx/sdrvjeQ4A3gLeFAAIUQAgRAGAEAUAQhQACFEAIEQBgBAFAEIUAIj/AAcbnBhl+orNAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "random_idx = np.random.randint(1000)\n",
    "\n",
    "image = x_test[random_idx]\n",
    "label = y_test[random_idx]\n",
    "\n",
    "\n",
    "visualize_image(image, label)\n",
    "input_layer = image.flatten()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "id": "3de34734",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 217,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_, new_output = forward_pass(input_layer, weights)\n",
    "norm_new_output = np.array(tf.nn.softmax(new_output)) # transforming the logits to probs\n",
    "np.argmax(norm_new_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76baf725",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.50433666, -0.00832172, -0.35964987,  1.45268183,  0.36296841,\n",
       "       -0.09697751,  0.24515608,  0.68653464, -0.15607574, -0.07038648])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "id": "43752579",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjUAAAHHCAYAAABHp6kXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAxDElEQVR4nO3deXyM5+L///ckZAQJUQkJsUQtpbU0Wl/UrpYGpS2qtGJrPRqnUcsp1VPV1naKKu1BtTht1Va0yqdIUU5bPvZ9r6WOWIskqCBz//7oz3w6DckEyZ25vJ6Px/045p5r7vs9c3JO3rnua2YclmVZAgAA8HF+dgcAAAC4Gyg1AADACJQaAABgBEoNAAAwAqUGAAAYgVIDAACMQKkBAABGoNQAAAAjUGoAAIARKDWAgd566y05HA67Y2Sr2NhYlSlTxu4YPqFMmTJq1aqV3TGAbEepAW7DjBkz5HA4tHHjxhw53+7du/XWW2/pyJEjOXI+3FqZMmXkcDjcW758+VS+fHkNHDhQ586dy9Esp06d0oABA1SpUiXlz59fBQoUUHR0tN59911duHAhR7PcwM8q7JTH7gAAMrd7924NGzZMDRs2ZHbi/zd16lS5XC5bzl29enX1799fknTlyhVt2rRJ48eP1+rVq7V+/focybBhwwY98cQTunjxorp06aLo6GhJ0saNGzVq1CitWbNGy5cvz5Esf8bPKuxEqQGQ4y5fvqz8+fPf0THy5s17l9JkXYkSJdSlSxf37Z49e6pgwYIaM2aMDhw4oPLly9/xOS5duqQCBQrc9L4LFy6oXbt28vf315YtW1SpUiWP+4cPH66pU6fecYasuHLligICAnL0nMBfcfkJyCZXr17Vm2++qejoaBUqVEgFChRQvXr1tGrVqnRjZ8+erejoaAUFBSk4OFgPPfSQPvjgA0l/XOpq3769JKlRo0buyx4//PBDlvJMnz5djRs3VlhYmJxOpypXrqxJkyZ5jOnatauKFi2qa9eupXt8s2bNVLFiRY99X3zxhaKjoxUYGKgiRYro2Wef1bFjxzzGNGzYUA8++KA2bdqk+vXrK3/+/Hr99ddvmnHMmDFyOBw6evRouvsGDx6sgIAAnT9/XlL6NTUNGzb0uCz0523GjBnevER3pHjx4pKkPHn+72/F7du3KzY2VlFRUcqXL5+KFy+u7t2767fffvN47I01ULt379Zzzz2nkJAQPfbYY7c815QpU3T8+HGNGzcuXaGRpGLFiumNN95It//HH3/Uo48+qnz58ikqKkqfffaZx/3nzp3TgAED9NBDD6lgwYIKDg5Wy5YttW3bNo9xP/zwgxwOh2bPnq033nhDJUqUUP78+TVhwoS78rMK3C5maoBskpycrE8++USdOnVSr169lJKSok8//VTNmzfX+vXrVb16dUlSQkKCOnXqpCZNmmj06NGSpD179uinn35SfHy86tevr1deeUUTJkzQ66+/rgceeECS3P/prUmTJqlKlSpq06aN8uTJo2+//VYvv/yyXC6X4uLiJEnPP/+8PvvsMy1btsxjYenJkye1cuVKDR061L1v+PDh+sc//qEOHTqoZ8+eOnPmjCZOnKj69etry5YtKly4sHvsb7/9ppYtW+rZZ59Vly5dVKxYsZtm7NChg/7+979r7ty5GjhwoMd9c+fOVbNmzRQSEnLTxw4ZMkQ9e/b02PfFF19o2bJlCgsLy9JrlZlr167p7Nmzkv6YodiyZYvGjRun+vXrq2zZsu5xCQkJOnTokLp166bixYtr165d+vjjj7Vr1y6tW7cu3WLu9u3bq3z58hoxYoQsy7rl+RctWqTAwEA988wzXmc+ePCgnnnmGfXo0UNdu3bVtGnTFBsbq+joaFWpUkWSdOjQIX399ddq3769ypYtq1OnTmnKlClq0KCBdu/erYiICI9jvvPOOwoICNCAAQOUmpqqZs2a3ZWfVeC2WQCybPr06ZYka8OGDbccc/36dSs1NdVj3/nz561ixYpZ3bt3d++Lj4+3goODrevXr9/yWPPmzbMkWatWrfIq39ChQ62//s/78uXL6cY1b97cioqKct9OS0uzSpYsaXXs2NFj3Lhx4yyHw2EdOnTIsizLOnLkiOXv728NHz7cY9yOHTusPHnyeOxv0KCBJcmaPHmyV9lr165tRUdHe+xbv369Jcn67LPP3Pu6du1qlS5d+pbH+emnn6y8efN6vNZ3Q+nSpS1J6ba6detaZ8+e9Rh7s9d81qxZliRrzZo17n03/vvq1KmTVxlCQkKsatWqZTnzn895+vRpy+l0Wv3793fvu3LlipWWlubx2MOHD1tOp9N6++233ftWrVplSbKioqLSPces/qwCdxOXn4Bs4u/v715j4HK5dO7cOV2/fl01a9bU5s2b3eMKFy6sS5cuKSEhIVvzBAYGuv+dlJSks2fPqkGDBjp06JCSkpIkSX5+furcubMWLVqklJQU9/iZM2eqTp067lmIBQsWyOVyqUOHDjp79qx7K168uMqXL5/uEpvT6VS3bt28ytmxY0dt2rRJv/zyi3vfnDlz5HQ69eSTT3p1jJMnT+qZZ55R9erV9a9//curx2RFrVq1lJCQoISEBC1evFjDhw/Xrl271KZNG/3+++/ucX9+za9cuaKzZ8/q//2//ydJHj8DN/Tu3dur8ycnJysoKChLmStXrqx69eq5b4eGhqpixYo6dOiQe5/T6ZSf3x+/FtLS0vTbb7+pYMGCqlix4k3zdu3a1eM5Anaj1ADZ6N///reqVq2qfPny6b777lNoaKiWLFniLhGS9PLLL6tChQpq2bKlSpYsqe7du2vp0qV3PctPP/2kpk2bqkCBAipcuLBCQ0Pda1v+nOeFF17Q77//roULF0qS9u3bp02bNun55593jzlw4IAsy1L58uUVGhrqse3Zs0enT5/2OHeJEiW8XkTavn17+fn5ac6cOZIky7I0b948tWzZUsHBwZk+/vr16+rQoYPS0tK0YMECOZ3ODMefOXNGJ0+edG8XL17M9BxFixZV06ZN1bRpU8XExOj111/XJ598op9//lmffPKJe9y5c+cUHx+vYsWKKTAwUKGhoe5i+OfX/IY/X7rKSHBwsEfp9EapUqXS7QsJCXGvUZL+KN/vv/++ypcvL6fTqaJFiyo0NFTbt2+/o7xATqHUANnkiy++UGxsrMqVK6dPP/1US5cuVUJCgho3buzxVuSwsDBt3bpVixYtUps2bbRq1Sq1bNlSXbt2vWtZfvnlFzVp0kRnz57VuHHjtGTJEiUkJOjVV1+VJI88lStXVnR0tL744gv38wgICFCHDh3cY1wulxwOh/s5/XWbMmWKx/mz8td8RESE6tWrp7lz50qS1q1bp19//VUdO3b06vEDBw7U2rVrNXfuXJUsWTLT8Y888ojCw8Pd25gxY7zO+mdNmjSRJK1Zs8a9r0OHDpo6dap69+6tBQsWaPny5e7CerO3o3v7OlWqVEn79+/X1atXvc7n7+9/0/3Wn9bujBgxQv369VP9+vXd65ESEhJUpUqVO8oL5BQWCgPZ5KuvvlJUVJQWLFjgsSD0z4ttbwgICFDr1q3VunVruVwuvfzyy5oyZYr+8Y9/6P7777/jTwf+9ttvlZqaqkWLFnn8xX6zd2JJf8zW9OvXTydOnNCXX36pmJgYjwW65cqVk2VZKlu2rCpUqHBH2W6mY8eOevnll7Vv3z7NmTNH+fPnV+vWrTN93OzZszV+/HiNHz9eDRo08OpcM2fO9LhkFBUVdVuZr1+/LknumZ7z589rxYoVGjZsmN588033uAMHDtzW8f+sdevWWrt2rebPn69OnTrd8fFu+Oqrr9SoUSN9+umnHvsvXLigokWLenUM0z/JGrkbMzVANrnxl/Gf/xL+3//9X61du9Zj3F/f3uvn56eqVatKklJTUyXJ/Xklt/spsTfLkpSUpOnTp990fKdOneRwOBQfH69Dhw55fCaLJD311FPy9/fXsGHD0r1Lx7KsdM8pq55++mn5+/tr1qxZmjdvnlq1anXLz2y5YefOnerZs6e6dOmi+Ph4r89Vt25d96Wkpk2b3nap+fbbbyVJ1apVk3Tz11ySxo8ff1vH/7PevXsrPDxc/fv31/79+9Pdf/r0ab377rtZPq6/v3+6vPPmzdPx48e9Psad/qwCd4KZGuAOTJs27abrX+Lj49WqVSstWLBA7dq1U0xMjA4fPqzJkyercuXKHus2evbsqXPnzqlx48YqWbKkjh49qokTJ6p69erut8JWr15d/v7+Gj16tJKSkuR0Ot2fOeONZs2auWeDXnrpJV28eFFTp05VWFiYTpw4kW58aGioWrRooXnz5qlw4cKKiYnxuL9cuXJ69913NXjwYB05ckRt27ZVUFCQDh8+rIULF+rFF1/UgAEDsvJSeggLC1OjRo00btw4paSkeHXp6cZC5BuXTv6sTp06t11Wbub48ePuc1y9elXbtm3TlClTVLRoUf3tb3+T9Me6l/r16+uf//ynrl27phIlSmj58uU6fPjwHZ8/JCRECxcu1BNPPKHq1at7fKLw5s2bNWvWLNWuXTvLx23VqpXefvttdevWTXXq1NGOHTs0c+bMLL12d/qzCtwRu952BfiyG2/pvtV27Ngxy+VyWSNGjLBKly5tOZ1Oq0aNGtbixYvTvRX5q6++spo1a2aFhYVZAQEBVqlSpayXXnrJOnHihMc5p06dakVFRVn+/v6ZvmX2Zm/pXrRokVW1alUrX758VpkyZazRo0db06ZNsyRZhw8fTneMuXPnWpKsF1988ZbnmT9/vvXYY49ZBQoUsAoUKGBVqlTJiouLs/bt2+ce06BBA6tKlSoZv6A3MXXqVEuSFRQUZP3+++/p7v/r63irt1pLsqZPn57l89/KX8/j5+dnhYWFWZ06dbIOHjzoMfa///2v1a5dO6tw4cJWoUKFrPbt21uJiYmWJGvo0KHucTf++zpz5kyWsiQmJlqvvvqqVaFCBStfvnxW/vz5rejoaGv48OFWUlKSR+aYmJh0j2/QoIHVoEED9+0rV65Y/fv3t8LDw63AwECrbt261tq1a9ONu/GW7nnz5t00V1Z+VoG7yWFZGXzCE4B71jfffKO2bdtqzZo1Hm8FBoDcilID4KZatWqlPXv26ODBgyz+BOATWFMDwMPs2bO1fft2LVmyRB988AGFBoDPYKYGgAeHw6GCBQuqY8eOmjx5sscXNAJAbsb/WwHwwN85AHwVn1MDAACMQKkBAABGuKcuP7lcLiUmJiooKIjFjwAA+AjLspSSkqKIiAj3N8nfzD1VahITExUZGWl3DAAAcBuOHTuW4RfV3lOlJigoSNIfL0pwcLDNaQAAgDeSk5MVGRnp/j1+K/dUqblxySk4OJhSAwCAj8ls6QgLhQEAgBEoNQAAwAiUGgAAYARKDQAAMAKlBgAAGIFSAwAAjECpAQAARqDUAAAAI1BqAACAESg1AADACJQaAABgBEoNAAAwAqUGAAAYgVIDAACMkMfuAAAAIOvKDFpid4R0joyKsfX8zNQAAAAjUGoAAIARKDUAAMAIlBoAAGAESg0AADACpQYAABiBUgMAAIxAqQEAAEag1AAAACNQagAAgBEoNQAAwAiUGgAAYARKDQAAMAKlBgAAGIFSAwAAjECpAQAARqDUAAAAI1BqAACAESg1AADACJQaAABgBEoNAAAwAqUGAAAYgVIDAACMQKkBAABGoNQAAAAjUGoAAIARKDUAAMAIlBoAAGAESg0AADACpQYAABiBUgMAAIxAqQEAAEag1AAAACNQagAAgBEoNQAAwAiUGgAAYARKDQAAMAKlBgAAGIFSAwAAjOAzpWbkyJF65JFHFBQUpLCwMLVt21b79u2zOxYAAMglfKbUrF69WnFxcVq3bp0SEhJ07do1NWvWTJcuXbI7GgAAyAXy2B3AW0uXLvW4PWPGDIWFhWnTpk2qX7++TakAAEBu4TOl5q+SkpIkSUWKFLnlmNTUVKWmprpvJycnZ3suAABgD5+5/PRnLpdLffv2Vd26dfXggw/ectzIkSNVqFAh9xYZGZmDKQEAQE7yyVITFxennTt3avbs2RmOGzx4sJKSktzbsWPHcighAADIaT53+alPnz5avHix1qxZo5IlS2Y41ul0yul05lAyAABgJ58pNZZl6W9/+5sWLlyoH374QWXLlrU7EgAAyEV8ptTExcXpyy+/1DfffKOgoCCdPHlSklSoUCEFBgbanA4AANjNZ9bUTJo0SUlJSWrYsKHCw8Pd25w5c+yOBgAAcgGfmamxLMvuCAAAIBfzmZkaAACAjFBqAACAESg1AADACJQaAABgBJ9ZKAzAPmUGLbE7QjpHRsXYHQFALsNMDQAAMAKlBgAAGIFSAwAAjECpAQAARqDUAAAAI1BqAACAESg1AADACJQaAABgBEoNAAAwAqUGAAAYgVIDAACMQKkBAABGoNQAAAAjUGoAAIARKDUAAMAIlBoAAGAESg0AADACpQYAABiBUgMAAIxAqQEAAEag1AAAACNQagAAgBEoNQAAwAiUGgAAYARKDQAAMAKlBgAAGIFSAwAAjECpAQAARqDUAAAAI1BqAACAESg1AADACJQaAABgBEoNAAAwAqUGAAAYgVIDAACMQKkBAABGoNQAAAAjUGoAAIARKDUAAMAIlBoAAGAESg0AADACpQYAABiBUgMAAIxAqQEAAEbIY3cAAIAZygxaYneEdI6Misl0jK/mRnrM1AAAACNQagAAgBEoNQAAwAiUGgAAYARKDQAAMILPlZqPPvpIZcqUUb58+VSrVi2tX7/e7kgAACAX8KlSM2fOHPXr109Dhw7V5s2bVa1aNTVv3lynT5+2OxoAALCZT5WacePGqVevXurWrZsqV66syZMnK3/+/Jo2bZrd0QAAgM18ptRcvXpVmzZtUtOmTd37/Pz81LRpU61du9bGZAAAIDfwmU8UPnv2rNLS0lSsWDGP/cWKFdPevXtv+pjU1FSlpqa6bycnJ2drRgAAYB+fKTW3Y+TIkRo2bFiOnMtXP2ab3HePybl99SPbffX19tXcvvpzQm5z+Mzlp6JFi8rf31+nTp3y2H/q1CkVL178po8ZPHiwkpKS3NuxY8dyIioAALCBz5SagIAARUdHa8WKFe59LpdLK1asUO3atW/6GKfTqeDgYI8NAACYyacuP/Xr109du3ZVzZo19eijj2r8+PG6dOmSunXrZnc0AABgM58qNR07dtSZM2f05ptv6uTJk6pevbqWLl2abvEwAAC49/hUqZGkPn36qE+fPnbHAAAAuYzPrKkBAADICKUGAAAYgVIDAACMQKkBAABGoNQAAAAjUGoAAIARKDUAAMAIlBoAAGAESg0AADACpQYAABiBUgMAAIxAqQEAAEag1AAAACNQagAAgBEoNQAAwAiUGgAAYARKDQAAMAKlBgAAGIFSAwAAjECpAQAARqDUAAAAI1BqAACAESg1AADACJQaAABgBEoNAAAwAqUGAAAYgVIDAACMQKkBAABGyGN3AOBecmRUjN0RAMBYzNQAAAAjUGoAAIARKDUAAMAIlBoAAGAESg0AADDCbZWa69ev6/vvv9eUKVOUkpIiSUpMTNTFixfvajgAAABvZfkt3UePHlWLFi3066+/KjU1VY8//riCgoI0evRopaamavLkydmREwAAIENZnqmJj49XzZo1df78eQUGBrr3t2vXTitWrLir4QAAALyV5Zma//znP/r5558VEBDgsb9MmTI6fvz4XQsGAACQFVmeqXG5XEpLS0u3/7///a+CgoLuSigAAICsynKpadasmcaPH+++7XA4dPHiRQ0dOlRPPPHE3cwGAADgtSxffho7dqyaN2+uypUr68qVK3ruued04MABFS1aVLNmzcqOjABwW/iuLeDekuVSU7JkSW3btk2zZ8/W9u3bdfHiRfXo0UOdO3f2WDgMAACQk27rW7rz5MmjLl263O0sAAAAty3Lpeazzz7L8P4XXnjhtsMAAADcriyXmvj4eI/b165d0+XLlxUQEKD8+fNTagAAgC2y/O6n8+fPe2wXL17Uvn379Nhjj7FQGAAA2OaufKFl+fLlNWrUqHSzOAAAADnlrn1Ld548eZSYmHi3DgcAAJAlWV5Ts2jRIo/blmXpxIkT+vDDD1W3bt27FgwAACArslxq2rZt63Hb4XAoNDRUjRs31tixY+9WLgAAgCzJcqlxuVzZkQMAAOCO3LU1NQAAAHbyaqamX79+Xh9w3Lhxtx0GAADgdnlVarZs2eLVwRwOxx2FAQAAuF1elZpVq1Zldw7YhG8xBgCYgjU1AADACLf1Ld0bN27U3Llz9euvv+rq1ase9y1YsOCuBPuzI0eO6J133tHKlSt18uRJRUREqEuXLhoyZIgCAgLu+vkAAIDvyfJMzezZs1WnTh3t2bNHCxcu1LVr17Rr1y6tXLlShQoVyo6M2rt3r1wul6ZMmaJdu3bp/fff1+TJk/X6669ny/kAAIDvyfJMzYgRI/T+++8rLi5OQUFB+uCDD1S2bFm99NJLCg8Pz46MatGihVq0aOG+HRUVpX379mnSpEkaM2ZMtpwTAAD4lizP1Pzyyy+KifljcWlAQIAuXbokh8OhV199VR9//PFdD3grSUlJKlKkSIZjUlNTlZyc7LEBAAAzZbnUhISEKCUlRZJUokQJ7dy5U5J04cIFXb58+e6mu4WDBw9q4sSJeumllzIcN3LkSBUqVMi9RUZG5kg+AACQ87wuNTfKS/369ZWQkCBJat++veLj49WrVy916tRJTZo0ydLJBw0aJIfDkeG2d+9ej8ccP35cLVq0UPv27dWrV68Mjz948GAlJSW5t2PHjmUpHwAA8B1er6mpWrWqHnnkEbVt21bt27eXJA0ZMkR58+bVzz//rKefflpvvPFGlk7ev39/xcbGZjgmKirK/e/ExEQ1atRIderU8epSl9PplNPpzFImAADgm7wuNatXr9b06dM1cuRIDR8+XE8//bR69uypQYMG3fbJQ0NDFRoa6tXY48ePq1GjRoqOjtb06dPl58dH7AAAgP/jdTOoV6+epk2bphMnTmjixIk6cuSIGjRooAoVKmj06NE6efJktoU8fvy4GjZsqFKlSmnMmDE6c+aMTp48ma3nBAAAviXL0x0FChRQt27dtHr1au3fv1/t27fXRx99pFKlSqlNmzbZkVEJCQk6ePCgVqxYoZIlSyo8PNy9AQAASHf4NQn333+/Xn/9db3xxhsKCgrSkiVL7lYuD7GxsbIs66YbAACAdJtfkyBJa9as0bRp0zR//nz5+fmpQ4cO6tGjx93MBgAA4LUslZrExETNmDFDM2bM0MGDB1WnTh1NmDBBHTp0UIECBbIrIwAAQKa8LjUtW7bU999/r6JFi+qFF15Q9+7dVbFixezMBgAA4DWvS03evHn11VdfqVWrVvL398/OTAAAAFnmdalZtGhRduYAAAC4I3yCHQAAMAKlBgAAGIFSAwAAjECpAQAARqDUAAAAI1BqAACAESg1AADACJQaAABghNv+QksAQPY4MirG7giAT2KmBgAAGIFSAwAAjECpAQAARqDUAAAAI1BqAACAESg1AADACJQaAABgBEoNAAAwAqUGAAAYgVIDAACMQKkBAABGoNQAAAAjUGoAAIARKDUAAMAIlBoAAGAESg0AADACpQYAABiBUgMAAIxAqQEAAEag1AAAACNQagAAgBEoNQAAwAiUGgAAYARKDQAAMAKlBgAAGIFSAwAAjECpAQAARqDUAAAAI1BqAACAESg1AADACJQaAABgBEoNAAAwAqUGAAAYgVIDAACMQKkBAABGoNQAAAAjUGoAAIARKDUAAMAIlBoAAGAESg0AADACpQYAABjB50pNamqqqlevLofDoa1bt9odBwAA5BI+V2r+/ve/KyIiwu4YAAAgl8ljd4Cs+O6777R8+XLNnz9f3333nd1xYKMjo2LsjgAAyGV8ptScOnVKvXr10tdff638+fN79ZjU1FSlpqa6bycnJ2dXPAAAYDOfuPxkWZZiY2PVu3dv1axZ0+vHjRw5UoUKFXJvkZGR2ZgSAADYydZSM2jQIDkcjgy3vXv3auLEiUpJSdHgwYOzdPzBgwcrKSnJvR07diybngkAALCbrZef+vfvr9jY2AzHREVFaeXKlVq7dq2cTqfHfTVr1lTnzp3173//+6aPdTqd6R4DAADMZGupCQ0NVWhoaKbjJkyYoHfffdd9OzExUc2bN9ecOXNUq1at7IwIAAB8hE8sFC5VqpTH7YIFC0qSypUrp5IlS9oRCQAA5DI+sVAYAAAgMz4xU/NXZcqUkWVZdscAAAC5CDM1AADACJQaAABgBEoNAAAwAqUGAAAYgVIDAACMQKkBAABGoNQAAAAjUGoAAIARKDUAAMAIlBoAAGAESg0AADACpQYAABiBUgMAAIxAqQEAAEag1AAAACNQagAAgBEoNQAAwAiUGgAAYARKDQAAMAKlBgAAGIFSAwAAjECpAQAARqDUAAAAI1BqAACAESg1AADACJQaAABgBEoNAAAwAqUGAAAYgVIDAACMQKkBAABGoNQAAAAjUGoAAIARKDUAAMAIlBoAAGAESg0AADACpQYAABiBUgMAAIxAqQEAAEag1AAAACNQagAAgBEoNQAAwAiUGgAAYARKDQAAMAKlBgAAGIFSAwAAjECpAQAARqDUAAAAI1BqAACAESg1AADACJQaAABgBEoNAAAwAqUGAAAYgVIDAACMQKkBAABGoNQAAAAj+FSpWbJkiWrVqqXAwECFhISobdu2dkcCAAC5RB67A3hr/vz56tWrl0aMGKHGjRvr+vXr2rlzp92xAABALuETpeb69euKj4/Xe++9px49erj3V65c2cZUAAAgN/GJy0+bN2/W8ePH5efnpxo1aig8PFwtW7bMdKYmNTVVycnJHhsAADCTT5SaQ4cOSZLeeustvfHGG1q8eLFCQkLUsGFDnTt37paPGzlypAoVKuTeIiMjcyoyAADIYbaWmkGDBsnhcGS47d27Vy6XS5I0ZMgQPf3004qOjtb06dPlcDg0b968Wx5/8ODBSkpKcm/Hjh3LqacGAABymK1ravr376/Y2NgMx0RFRenEiROSPNfQOJ1ORUVF6ddff73lY51Op5xO513JCgAAcjdbS01oaKhCQ0MzHRcdHS2n06l9+/bpsccekyRdu3ZNR44cUenSpbM7JgAA8AE+8e6n4OBg9e7dW0OHDlVkZKRKly6t9957T5LUvn17m9MBAIDcwCdKjSS99957ypMnj55//nn9/vvvqlWrllauXKmQkBC7owEAgFzAZ0pN3rx5NWbMGI0ZM8buKAAAIBfyibd0AwAAZIZSAwAAjECpAQAARqDUAAAAI1BqAACAESg1AADACJQaAABgBEoNAAAwAqUGAAAYgVIDAACMQKkBAABGoNQAAAAjUGoAAIARKDUAAMAIlBoAAGAESg0AADACpQYAABiBUgMAAIxAqQEAAEag1AAAACPksTuAKY6MirE7AgAA9zRmagAAgBEoNQAAwAiUGgAAYARKDQAAMAKlBgAAGIFSAwAAjECpAQAARqDUAAAAI1BqAACAESg1AADACJQaAABgBEoNAAAwAqUGAAAYgVIDAACMQKkBAABGyGN3gJxkWZYkKTk52eYkAADAWzd+b9/4PX4r91SpSUlJkSRFRkbanAQAAGRVSkqKChUqdMv7HVZmtccgLpdLiYmJCgoKksPhsDvOTSUnJysyMlLHjh1TcHCw3XG8Ru6cRe6cRe6cRe6c5Qu5LctSSkqKIiIi5Od365Uz99RMjZ+fn0qWLGl3DK8EBwfn2h+ujJA7Z5E7Z5E7Z5E7Z+X23BnN0NzAQmEAAGAESg0AADACpSaXcTqdGjp0qJxOp91RsoTcOYvcOYvcOYvcOctXc9/MPbVQGAAAmIuZGgAAYARKDQAAMAKlBgAAGIFSAwAAjECpyUU++ugjlSlTRvny5VOtWrW0fv16uyNlas2aNWrdurUiIiLkcDj09ddf2x0pUyNHjtQjjzyioKAghYWFqW3bttq3b5/dsTI1adIkVa1a1f0BWbVr19Z3331nd6wsGzVqlBwOh/r27Wt3lAy99dZbcjgcHlulSpXsjuWV48ePq0uXLrrvvvsUGBiohx56SBs3brQ7VqbKlCmT7jV3OByKi4uzO9otpaWl6R//+IfKli2rwMBAlStXTu+8806m31GUG6SkpKhv374qXbq0AgMDVadOHW3YsMHuWHeEUpNLzJkzR/369dPQoUO1efNmVatWTc2bN9fp06ftjpahS5cuqVq1avroo4/sjuK11atXKy4uTuvWrVNCQoKuXbumZs2a6dKlS3ZHy1DJkiU1atQobdq0SRs3blTjxo315JNPateuXXZH89qGDRs0ZcoUVa1a1e4oXqlSpYpOnDjh3n788Ue7I2Xq/Pnzqlu3rvLmzavvvvtOu3fv1tixYxUSEmJ3tExt2LDB4/VOSEiQJLVv397mZLc2evRoTZo0SR9++KH27Nmj0aNH65///KcmTpxod7RM9ezZUwkJCfr888+1Y8cONWvWTE2bNtXx48ftjnb7LOQKjz76qBUXF+e+nZaWZkVERFgjR460MVXWSLIWLlxod4wsO336tCXJWr16td1RsiwkJMT65JNP7I7hlZSUFKt8+fJWQkKC1aBBAys+Pt7uSBkaOnSoVa1aNbtjZNlrr71mPfbYY3bHuCvi4+OtcuXKWS6Xy+4otxQTE2N1797dY99TTz1lde7c2aZE3rl8+bLl7+9vLV682GP/ww8/bA0ZMsSmVHeOmZpc4OrVq9q0aZOaNm3q3ufn56emTZtq7dq1Nia7NyQlJUmSihQpYnMS76WlpWn27Nm6dOmSateubXccr8TFxSkmJsbj5zy3O3DggCIiIhQVFaXOnTvr119/tTtSphYtWqSaNWuqffv2CgsLU40aNTR16lS7Y2XZ1atX9cUXX6h79+659guIJalOnTpasWKF9u/fL0natm2bfvzxR7Vs2dLmZBm7fv260tLSlC9fPo/9gYGBPjEjeSv31Bda5lZnz55VWlqaihUr5rG/WLFi2rt3r02p7g0ul0t9+/ZV3bp19eCDD9odJ1M7duxQ7dq1deXKFRUsWFALFy5U5cqV7Y6VqdmzZ2vz5s0+db2+Vq1amjFjhipWrKgTJ05o2LBhqlevnnbu3KmgoCC7493SoUOHNGnSJPXr10+vv/66NmzYoFdeeUUBAQHq2rWr3fG89vXXX+vChQuKjY21O0qGBg0apOTkZFWqVEn+/v5KS0vT8OHD1blzZ7ujZSgoKEi1a9fWO++8owceeEDFihXTrFmztHbtWt1///12x7ttlBrc0+Li4rRz506f+cukYsWK2rp1q5KSkvTVV1+pa9euWr16da4uNseOHVN8fLwSEhLS/VWYm/35L+2qVauqVq1aKl26tObOnasePXrYmCxjLpdLNWvW1IgRIyRJNWrU0M6dOzV58mSfKjWffvqpWrZsqYiICLujZGju3LmaOXOmvvzyS1WpUkVbt25V3759FRERketf788//1zdu3dXiRIl5O/vr4cfflidOnXSpk2b7I522yg1uUDRokXl7++vU6dOeew/deqUihcvblMq8/Xp00eLFy/WmjVrVLJkSbvjeCUgIMD9V1R0dLQ2bNigDz74QFOmTLE52a1t2rRJp0+f1sMPP+zel5aWpjVr1ujDDz9Uamqq/P39bUzoncKFC6tChQo6ePCg3VEyFB4enq7kPvDAA5o/f75NibLu6NGj+v7777VgwQK7o2Rq4MCBGjRokJ599llJ0kMPPaSjR49q5MiRub7UlCtXTqtXr9alS5eUnJys8PBwdezYUVFRUXZHu22sqckFAgICFB0drRUrVrj3uVwurVixwmfWS/gSy7LUp08fLVy4UCtXrlTZsmXtjnTbXC6XUlNT7Y6RoSZNmmjHjh3aunWre6tZs6Y6d+6srVu3+kShkaSLFy/ql19+UXh4uN1RMlS3bt10H1Gwf/9+lS5d2qZEWTd9+nSFhYUpJibG7iiZunz5svz8PH+V+vv7y+Vy2ZQo6woUKKDw8HCdP39ey5Yt05NPPml3pNvGTE0u0a9fP3Xt2lU1a9bUo48+qvHjx+vSpUvq1q2b3dEydPHiRY+/XA8fPqytW7eqSJEiKlWqlI3Jbi0uLk5ffvmlvvnmGwUFBenkyZOSpEKFCikwMNDmdLc2ePBgtWzZUqVKlVJKSoq+/PJL/fDDD1q2bJnd0TIUFBSUbr1SgQIFdN999+XqdUwDBgxQ69atVbp0aSUmJmro0KHy9/dXp06d7I6WoVdffVV16tTRiBEj1KFDB61fv14ff/yxPv74Y7ujecXlcmn69Onq2rWr8uTJ/b+iWrdureHDh6tUqVKqUqWKtmzZonHjxql79+52R8vUsmXLZFmWKlasqIMHD2rgwIGqVKlSrv+9kyG7336F/zNx4kSrVKlSVkBAgPXoo49a69atsztSplatWmVJSrd17drV7mi3dLO8kqzp06fbHS1D3bt3t0qXLm0FBARYoaGhVpMmTazly5fbHeu2+MJbujt27GiFh4dbAQEBVokSJayOHTtaBw8etDuWV7799lvrwQcftJxOp1WpUiXr448/tjuS15YtW2ZJsvbt22d3FK8kJydb8fHxVqlSpax8+fJZUVFR1pAhQ6zU1FS7o2Vqzpw5VlRUlBUQEGAVL17ciouLsy5cuGB3rDvisCwf+NhDAACATLCmBgAAGIFSAwAAjECpAQAARqDUAAAAI1BqAACAESg1AADACJQaAABgBEoNAAAwAqUGgKQ/vk27e/fuioiIUEBAgEqXLq34+Hj99ttvWTrOkSNH5HA4tHXr1mzJ6XA49PXXX2fLsQH4NkoNAB06dEg1a9bUgQMHNGvWLB08eFCTJ092f6nquXPn7I7oM65du2Z3BOCeRakBoLi4OAUEBGj58uVq0KCBSpUqpZYtW+r777/X8ePHNWTIEPfYm82UFC5cWDNmzJAk97ee16hRQw6HQw0bNpQkxcbGqm3btho2bJhCQ0MVHBys3r176+rVq+7jlClTRuPHj/c4dvXq1fXWW2+575ekdu3ayeFwuG//1Y3ZogULFqhRo0bKnz+/qlWrprVr13qM+/HHH1WvXj0FBgYqMjJSr7zyii5duuT1c71xnjlz5qhBgwbKly+fZs6cKZfLpbffflslS5aU0+lU9erVtXTp0iznA5A1lBrgHnfu3DktW7ZML7/8crpvKS9evLg6d+6sOXPmyNuviVu/fr0k6fvvv9eJEye0YMEC930rVqzQnj179MMPP2jWrFlasGCBhg0b5nXWDRs2SJKmT5+uEydOuG/fypAhQzRgwABt3bpVFSpUUKdOnXT9+nVJ0i+//KIWLVro6aef1vbt2zVnzhz9+OOP6tOnj9d5bhg0aJDi4+O1Z88eNW/eXB988IHGjh2rMWPGaPv27WrevLnatGmjAwcOeJ0PQNZRaoB73IEDB2RZlh544IGb3v/AAw/o/PnzOnPmjFfHCw0NlSTdd999Kl68uIoUKeK+LyAgQNOmTVOVKlUUExOjt99+WxMmTJDL5crSsQsXLqzixYu7b9/KgAEDFBMTowoVKmjYsGE6evSoDh48KEkaOXKkOnfurL59+6p8+fKqU6eOJkyYoM8++0xXrlzxKs8Nffv21VNPPaWyZcsqPDxcY8aM0WuvvaZnn31WFStW1OjRo1W9evV0s1AZ5QOQdZQaAJLk9UzMnahWrZry58/vvl27dm1dvHhRx44dy5bzVa1a1f3v8PBwSdLp06clSdu2bdOMGTNUsGBB99a8eXO5XC4dPnw4S+epWbOm+9/JyclKTExU3bp1PcbUrVtXe/bs8TofgKzLY3cAAPa6//775XA4tGfPHrVr1y7d/Xv27FFISIh7VsThcKQrQHdrcayfn99dPXbevHnd/3Y4HJLknhW6ePGiXnrpJb3yyivpHleqVCn3Y7zJU6BAgbueD0DWMVMD3OPuu+8+Pf744/rXv/6l33//3eO+kydPaubMmerYsaP7l25oaKhOnDjhHnPgwAFdvnzZfTsgIECSlJaWlu5c27Zt8zjHunXrVLBgQUVGRt702MnJyelmTfLmzXvTY2fVww8/rN27d+v+++9Pt914Dpk915sJDg5WRESEfvrpJ4/9P/30kypXrnzHuQHcGqUGgD788EOlpqaqefPmWrNmjY4dO6alS5fq8ccfV4kSJTR8+HD32MaNG+vDDz/Uli1btHHjRvXu3dtjxiEsLEyBgYFaunSpTp06paSkJPd9V69eVY8ePbR79279z//8j4YOHao+ffrIz8/PfezPP/9c//nPf7Rjxw517dpV/v7+HlnLlCmjFStW6OTJkzp//vxtP+fXXntNP//8s/r06aOtW7fqwIED+uabbzwWCmf2XG9l4MCBGj16tObMmaN9+/Zp0KBB2rp1q+Lj4287L4DMUWoAqHz58tq4caOioqLUoUMHlStXTi+++KIaNWqktWvXeiz2HTt2rCIjI1WvXj0999xzGjBggMc6mTx58mjChAmaMmWKIiIi9OSTT7rva9KkicqXL6/69eurY8eOatOmjfvt2pI0ePBgNWjQQK1atVJMTIzatm2rcuXKeWQdO3asEhISFBkZqRo1atz2c65atapWr16t/fv3q169eqpRo4befPNNRUREeP1cb+WVV15Rv3791L9/fz300ENaunSpFi1apPLly992XgCZc1g5sToQwD0vNjZWFy5c4NOAAWQbZmoAAIARKDUAAMAIXH4CAABGYKYGAAAYgVIDAACMQKkBAABGoNQAAAAjUGoAAIARKDUAAMAIlBoAAGAESg0AADACpQYAABjh/wO8AuLMkGbzEwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "x_indices = range(10)\n",
    "\n",
    "plt.bar(x_indices, new_output)\n",
    "plt.xlabel('Output neuron')\n",
    "plt.ylabel('Value')\n",
    "plt.title('Last layer viz - Bar Chart')\n",
    "\n",
    "# Customize x-axis tick labels\n",
    "plt.xticks(x_indices, x_indices)\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee95cfc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import os\n",
    "\n",
    "root = os.getcwd()\n",
    "\n",
    "with open(os.path.join(root  ,'new_trained_weights.pkl'), 'wb') as file:\n",
    "    pickle.dump(weights, file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54234e3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(root  ,'trained_weights.pkl'), 'rb') as file:\n",
    "    var = pickle.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5ffc867",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}