02_DL0101EN-1-1-Forward-Propagation-py-v1 0__1__M.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://cognitiveclass.ai/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDL0101ENSkillsNetwork945-2022-01-01\"><img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0101EN-SkillsNetwork/images/IDSN-logo.png\" width=\"400\"> </a>\n",
    "\n",
    "<h1 align=center><font size = 5>Artificial Neural Networks - Forward Propagation</font></h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, we will build a neural network from scratch and code how it performs predictions using forward propagation. Please note that all deep learning libraries have the entire training and prediction processes implemented, and so in practice you wouldn't really need to build a neural network from scratch. However, hopefully completing this lab will help you understand neural networks and how they work even better.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Artificial Neural Networks - Forward Propagation</h2>\n",
    "\n",
    "<h3>Objective for this Notebook<h3>    \n",
    "<h5> 1. Initalize a Network</h5>\n",
    "<h5> 2. Compute Weighted Sum at Each Node. </h5>\n",
    "<h5> 3. Compute Node Activation </h5>\n",
    "<h5> 4. Access your <b>Flask</b> app via a webpage anywhere using a custom link. </h5>     \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "\n",
    "<font size = 3>    \n",
    "\n",
    "1. <a href=\"#item11\">Recap</a>\n",
    "2. <a href=\"#item12\">Initalize a Network</a>  \n",
    "3. <a href=\"#item13\">Compute Weighted Sum at Each Node</a>  \n",
    "4. <a href=\"#item14\">Compute Node Activation</a>  \n",
    "5. <a href=\"#item15\">Forward Propagation</a>\n",
    "\n",
    "</font>\n",
    "    \n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"item1\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='item11'></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recap\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the videos, let's recap how a neural network makes predictions through the forward propagation process. Here is a neural network that takes two inputs, has one hidden layer with two nodes, and an output layer with one node.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://cocl.us/neural_network_example\" alt=\"Neural Network Example\" width=\"600px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by randomly initializing the weights and the biases in the network. We have 6 weights and 3 biases, one for each node in the hidden layer as well as for each node in the output layer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# All Libraries required for this lab are listed below. The libraries pre-installed on Skills Network Labs are commented. \n",
    "# If you run this notebook on a different environment, e.g. your desktop, you may need to uncomment and install certain libraries.\n",
    "\n",
    "#!pip install numpy==1.21.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np # import Numpy library to generate \n",
    "\n",
    "weights = np.around(np.random.uniform(size=6), decimals=2) # initialize the weights\n",
    "biases = np.around(np.random.uniform(size=3), decimals=2) # initialize the biases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print the weights and biases for sanity check.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.84 0.14 0.8  0.52 0.91 0.9 ]\n",
      "[0.67 0.87 0.17]\n"
     ]
    }
   ],
   "source": [
    "print(weights)\n",
    "print(biases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the weights and the biases defined for the network, let's compute the output for a given input, $x_1$ and $x_2$.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x1 is 0.5 and x2 is 0.85\n"
     ]
    }
   ],
   "source": [
    "x_1 = 0.5 # input 1\n",
    "x_2 = 0.85 # input 2\n",
    "\n",
    "print('x1 is {} and x2 is {}'.format(x_1, x_2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by computing the wighted sum of the inputs, $z_{1, 1}$, at the first node of the hidden layer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The weighted sum of the inputs at the first node in the hidden layer is 1.209\n"
     ]
    }
   ],
   "source": [
    "z_11 = x_1 * weights[0] + x_2 * weights[1] + biases[0]\n",
    "\n",
    "print('The weighted sum of the inputs at the first node in the hidden layer is {}'.format(z_11))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's compute the weighted sum of the inputs, $z_{1, 2}$, at the second node of the hidden layer. Assign the value to **z_12**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "z_12 = x_1 * weights[2] + x_2 * weights[3] + biases[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "z_12 = x_1 * weights[2] + x_2 * weights[3] + biases[1]\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the weighted sum.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The weighted sum of the inputs at the second node in the hidden layer is 1.712\n"
     ]
    }
   ],
   "source": [
    "print('The weighted sum of the inputs at the second node in the hidden layer is {}'.format(np.around(z_12, decimals=4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, assuming a sigmoid activation function, let's compute the activation of the first node, $a_{1, 1}$, in the hidden layer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The activation of the first node in the hidden layer is 0.7701\n"
     ]
    }
   ],
   "source": [
    "a_11 = 1.0 / (1.0 + np.exp(-z_11))\n",
    "\n",
    "print('The activation of the first node in the hidden layer is {}'.format(np.around(a_11, decimals=4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also compute the activation of the second node, $a_{1, 2}$, in the hidden layer. Assign the value to **a_12**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "a_12 = 1.0 / (1.0 + np.exp(-z_12))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "a_12 = 1.0 / (1.0 + np.exp(-z_12))\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the activation of the second node.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The activation of the second node in the hidden layer is 0.8471\n"
     ]
    }
   ],
   "source": [
    "print('The activation of the second node in the hidden layer is {}'.format(np.around(a_12, decimals=4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now these activations will serve as the inputs to the output layer. So, let's compute the weighted sum of these inputs to the node in the output layer. Assign the value to **z_2**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "z_2 = a_11 * weights[4] + a_12 * weights[5] + biases[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "z_2 = a_11 * weights[4] + a_12 * weights[5] + biases[2]\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the weighted sum of the inputs at the node in the output layer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The weighted sum of the inputs at the node in the output layer is 1.6332\n"
     ]
    }
   ],
   "source": [
    "print('The weighted sum of the inputs at the node in the output layer is {}'.format(np.around(z_2, decimals=4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's compute the output of the network as the activation of the node in the output layer. Assign the value to **a_2**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "a_2 = 1.0 / (1.0 + np.exp(-z_2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "a_2 = 1.0 / (1.0 + np.exp(-z_2))\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the activation of the node in the output layer which is equivalent to the prediction made by the network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The output of the network for x1 = 0.5 and x2 = 0.85 is 0.8366\n"
     ]
    }
   ],
   "source": [
    "print('The output of the network for x1 = 0.5 and x2 = 0.85 is {}'.format(np.around(a_2, decimals=4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obviously, neural networks for real problems are composed of many hidden layers and many more nodes in each layer. So, we can't continue making predictions using this very inefficient approach of computing the weighted sum at each node and the activation of each node manually. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to code an automatic way of making predictions, let's generalize our network. A general network would take $n$ inputs, would have many hidden layers, each hidden layer having $m$ nodes, and would have an output layer. Although the network is showing one hidden layer, but we will code the network to have many hidden layers. Similarly, although the network shows an output layer with one node, we will code the network to have more than one node in the output layer.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://cocl.us/general_neural_network\" alt=\"Neural Network General\" width=\"600px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"item2\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='item12'></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize a Network\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by formally defining the structure of the network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 2 # number of inputs\n",
    "num_hidden_layers = 2 # number of hidden layers\n",
    "m = [2, 2] # number of nodes in each hidden layer\n",
    "num_nodes_output = 1 # number of nodes in the output layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we defined the structure of the network, let's go ahead and inititailize the weights and the biases in the network to random numbers. In order to be able to initialize the weights and the biases to random numbers, we will need to import the **Numpy** library.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'layer_1': {'node_1': {'weights': array([0.63, 0.63]), 'bias': array([0.48])}, 'node_2': {'weights': array([0.48, 0.96]), 'bias': array([0.93])}}, 'layer_2': {'node_1': {'weights': array([0.33, 0.83]), 'bias': array([0.59])}, 'node_2': {'weights': array([0.11, 0.18]), 'bias': array([0.7])}}, 'output': {'node_1': {'weights': array([0.05, 0.54]), 'bias': array([0.33])}}}\n"
     ]
    }
   ],
   "source": [
    "import numpy as np # import the Numpy library\n",
    "\n",
    "num_nodes_previous = n # number of nodes in the previous layer\n",
    "\n",
    "network = {} # initialize network an an empty dictionary\n",
    "\n",
    "# loop through each layer and randomly initialize the weights and biases associated with each node\n",
    "# notice how we are adding 1 to the number of hidden layers in order to include the output layer\n",
    "for layer in range(num_hidden_layers + 1): \n",
    "    \n",
    "    # determine name of layer\n",
    "    if layer == num_hidden_layers:\n",
    "        layer_name = 'output'\n",
    "        num_nodes = num_nodes_output\n",
    "    else:\n",
    "        layer_name = 'layer_{}'.format(layer + 1)\n",
    "        num_nodes = m[layer]\n",
    "    \n",
    "    # initialize weights and biases associated with each node in the current layer\n",
    "    network[layer_name] = {}\n",
    "    for node in range(num_nodes):\n",
    "        node_name = 'node_{}'.format(node+1)\n",
    "        network[layer_name][node_name] = {\n",
    "            'weights': np.around(np.random.uniform(size=num_nodes_previous), decimals=2),\n",
    "            'bias': np.around(np.random.uniform(size=1), decimals=2),\n",
    "        }\n",
    "    \n",
    "    num_nodes_previous = num_nodes\n",
    "    \n",
    "print(network) # print network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Awesome! So now with the above code, we are able to initialize the weights and the biases pertaining to any network of any number of hidden layers and number of nodes in each layer. But let's put this code in a function so that we are able to repetitively execute all this code whenever we want to construct a neural network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initialize_network(num_inputs, num_hidden_layers, num_nodes_hidden, num_nodes_output):\n",
    "    \n",
    "    num_nodes_previous = num_inputs # number of nodes in the previous layer\n",
    "\n",
    "    network = {}\n",
    "    \n",
    "    # loop through each layer and randomly initialize the weights and biases associated with each layer\n",
    "    for layer in range(num_hidden_layers + 1):\n",
    "        \n",
    "        if layer == num_hidden_layers:\n",
    "            layer_name = 'output' # name last layer in the network output\n",
    "            num_nodes = num_nodes_output\n",
    "        else:\n",
    "            layer_name = 'layer_{}'.format(layer + 1) # otherwise give the layer a number\n",
    "            num_nodes = num_nodes_hidden[layer] \n",
    "        \n",
    "        # initialize weights and bias for each node\n",
    "        network[layer_name] = {}\n",
    "        for node in range(num_nodes):\n",
    "            node_name = 'node_{}'.format(node+1)\n",
    "            network[layer_name][node_name] = {\n",
    "                'weights': np.around(np.random.uniform(size=num_nodes_previous), decimals=2),\n",
    "                'bias': np.around(np.random.uniform(size=1), decimals=2),\n",
    "            }\n",
    "    \n",
    "        num_nodes_previous = num_nodes\n",
    "\n",
    "    return network # return the network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the *initialize_network* function to create a network that:\n",
    "\n",
    "1. takes 5 inputs\n",
    "2. has three hidden layers\n",
    "3. has 3 nodes in the first layer, 2 nodes in the second layer, and 3 nodes in the third layer\n",
    "4. has 1 node in the output layer\n",
    "\n",
    "Call the network **small_network**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "small_network = initialize_network(5, 3, [3, 2, 3], 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "small_network = initialize_network(5, 3, [3, 2, 3], 1)\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"item3\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='item13'></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute Weighted Sum at Each Node\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The weighted sum at each node is computed as the dot product of the inputs and the weights plus the bias. So let's create a function called *compute_weighted_sum* that does just that.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_weighted_sum(inputs, weights, bias):\n",
    "    return np.sum(inputs * weights) + bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's generate 5 inputs that we can feed to **small_network**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The inputs to the network are [0.15 0.74 0.26 0.53 0.01]\n"
     ]
    }
   ],
   "source": [
    "from random import seed\n",
    "import numpy as np\n",
    "\n",
    "np.random.seed(12)\n",
    "inputs = np.around(np.random.uniform(size=5), decimals=2)\n",
    "\n",
    "print('The inputs to the network are {}'.format(inputs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the *compute_weighted_sum* function to compute the weighted sum at the first node in the first hidden layer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The weighted sum at the first node in the hidden layer is 1.3978\n"
     ]
    }
   ],
   "source": [
    "### type your answer here\n",
    "\n",
    "node_weights = small_network['layer_1']['node_1']['weights']\n",
    "node_bias = small_network['layer_1']['node_1']['bias']\n",
    "\n",
    "weighted_sum = compute_weighted_sum(inputs, node_weights, node_bias)\n",
    "print('The weighted sum at the first node in the hidden layer is {}'.format(np.around(weighted_sum[0], decimals=4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "node_weights = small_network['layer_1']['node_1']['weights']\n",
    "node_bias = small_network['layer_1']['node_1']['bias']\n",
    "\n",
    "weighted_sum = compute_weighted_sum(inputs, node_weights, node_bias)\n",
    "print('The weighted sum at the first node in the hidden layer is {}'.format(np.around(weighted_sum[0], decimals=4)))\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"item4\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='item14'></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute Node Activation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the output of each node is simply a non-linear tranformation of the weighted sum. We use activation functions for this mapping. Let's use the sigmoid function as the activation function here. So let's define a function that takes a weighted sum as input and returns the non-linear transformation of the input using the sigmoid function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def node_activation(weighted_sum):\n",
    "    return 1.0 / (1.0 + np.exp(-1 * weighted_sum))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the *node_activation* function to compute the output of the first node in the first hidden layer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The output of the first node in the hidden layer is 0.8018\n"
     ]
    }
   ],
   "source": [
    "### type your answer here\n",
    "\n",
    "node_output  = node_activation(compute_weighted_sum(inputs, node_weights, node_bias))\n",
    "print('The output of the first node in the hidden layer is {}'.format(np.around(node_output[0], decimals=4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "node_output  = node_activation(compute_weighted_sum(inputs, node_weights, node_bias))\n",
    "print('The output of the first node in the hidden layer is {}'.format(np.around(node_output[0], decimals=4)))\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"item5\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='item15'></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Forward Propagation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final piece of building a neural network that can perform predictions is to put everything together. So let's create a function that applies the *compute_weighted_sum* and *node_activation* functions to each node in the network and propagates the data all the way to the output layer and outputs a prediction for each node in the output layer.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The way we are going to accomplish this is through the following procedure:\n",
    "\n",
    "1. Start with the input layer as the input to the first hidden layer.\n",
    "2. Compute the weighted sum at the nodes of the current layer.\n",
    "3. Compute the output of the nodes of the current layer.\n",
    "4. Set the output of the current layer to be the input to the next layer.\n",
    "5. Move to the next layer in the network.\n",
    "5. Repeat steps 2 - 4 until we compute the output of the output layer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_propagate(network, inputs):\n",
    "    \n",
    "    layer_inputs = list(inputs) # start with the input layer as the input to the first hidden layer\n",
    "    \n",
    "    for layer in network:\n",
    "        \n",
    "        layer_data = network[layer]\n",
    "        \n",
    "        layer_outputs = [] \n",
    "        for layer_node in layer_data:\n",
    "        \n",
    "            node_data = layer_data[layer_node]\n",
    "        \n",
    "            # compute the weighted sum and the output of each node at the same time \n",
    "            node_output = node_activation(compute_weighted_sum(layer_inputs, node_data['weights'], node_data['bias']))\n",
    "            layer_outputs.append(np.around(node_output[0], decimals=4))\n",
    "            \n",
    "        if layer != 'output':\n",
    "            print('The outputs of the nodes in hidden layer number {} is {}'.format(layer.split('_')[1], layer_outputs))\n",
    "    \n",
    "        layer_inputs = layer_outputs # set the output of this layer to be the input to next layer\n",
    "\n",
    "    network_predictions = layer_outputs\n",
    "    return network_predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the *forward_propagate* function to compute the prediction of our small network\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The outputs of the nodes in hidden layer number 1 is [0.8018, 0.8836, 0.7253]\n",
      "The outputs of the nodes in hidden layer number 2 is [0.7806, 0.8317]\n",
      "The outputs of the nodes in hidden layer number 3 is [0.7641, 0.7645, 0.8609]\n",
      "The predicted value by the network for the given input is 0.7929\n"
     ]
    }
   ],
   "source": [
    "### type your answser here\n",
    "\n",
    "predictions = forward_propagate(small_network, inputs)\n",
    "print('The predicted value by the network for the given input is {}'.format(np.around(predictions[0], decimals=4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "predictions = forward_propagate(small_network, inputs)\n",
    "print('The predicted value by the network for the given input is {}'.format(np.around(predictions[0], decimals=4)))\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we built the code to define a neural network. We can specify the number of inputs that a neural network can take, the number of hidden layers as well as the number of nodes in each hidden layer, and the number of nodes in the output layer.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first use the *initialize_network* to create our neural network and define its weights and biases.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_network = initialize_network(5, 3, [2, 3, 2], 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, for a given input,\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = np.around(np.random.uniform(size=5), decimals=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we compute the network predictions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The outputs of the nodes in hidden layer number 1 is [0.8857, 0.8889]\n",
      "The outputs of the nodes in hidden layer number 2 is [0.7822, 0.6965, 0.7411]\n",
      "The outputs of the nodes in hidden layer number 3 is [0.868, 0.881]\n",
      "The predicted values by the network for the given input are [0.8952, 0.8222, 0.8035]\n"
     ]
    }
   ],
   "source": [
    "predictions = forward_propagate(my_network, inputs)\n",
    "print('The predicted values by the network for the given input are {}'.format(predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to play around with the code by creating different networks of different structures and enjoy making predictions using the *forward_propagate* function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "### create a network\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "### create another network\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thank you for completing this lab!\n",
    "\n",
    "This notebook was created by [Alex Aklson](https://www.linkedin.com/in/aklson/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDL0101ENSkillsNetwork945-2022-01-01). I hope you found this lab interesting and educational. Feel free to contact me if you have any questions!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Change Log\n",
    "\n",
    "|  Date (YYYY-MM-DD) |  Version | Changed By  |  Change Description |\n",
    "|---|---|---|---|\n",
    "| 2020-09-21  | 2.0  | Srishti  |  Migrated Lab to Markdown and added to course repo in GitLab |\n",
    "\n",
    "\n",
    "\n",
    "<hr>\n",
    "\n",
    "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is part of a course on **Coursera** called *Introduction to Deep Learning & Neural Networks with Keras*. If you accessed this notebook outside the course, you can take this course online by clicking [here](https://cocl.us/DL0101EN_Coursera_Week1_LAB1).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "Copyright &copy; 2019 [IBM Developer Skills Network](https://cognitiveclass.ai/?utm_medium=dswb&utm_source=bducopyrightlink&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDL0101ENSkillsNetwork945-2022-01-01&utm_campaign=bdu). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDL0101ENSkillsNetwork945-2022-01-01).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}