Neural Network form scratch (MNIST)

This Repo creates a Neural Network form scratch using only numpy library. The NN is tested on MNIST-Digits dataset which contains 10 classes for classification. MNIST like a 'hello_world' dataset in machine learning community. This repo also builds the popular optimizers like;

• ADAM
• RMSprop
• Adagrad
• Gradient descent with and without momentum term by using only numpy The effect of different activations functions (e.g. sigmoid, ReLu) is also studied in this repo.

This repo takes you through the steps regarding what is happening under the hood of artificial neural networks created by high level libraries like

• Tenforflow
• Keras
• Pytorch

As this repo only uses numpy so it only runs on CPU that's why its considerably slower than the models running on GPUs.

Dependencies

1. numpy
2. tqdm
3. scipy
4. matlab
5. idx2numpy

Building the Neural Network

If you are completely new to machine learning I recommend you to read this book along side this repo. I'll be using a lot of concepts discussesd in the book above and also in cs231.

Neural Network Architecture

Training with Sigmoid activation

Follwing is the architecture of the neural network constructed in the scripts.

You can study the details and math behind the neural network in the book above, here I'll explain things via presentation style images. Following is actually a summary of what you'll study in above mentioned lectures

Let's first look at a single neuron(with sigmoid)

Now, let's look at sigmoid function and how to impliment in python.

Now we can write the code for forward propagation.

The following images summarize how the backpropagation works in neural networks and how the loss function back propagates the loss and how the gradients are calculated.

Finally now we can update the weights and biases via following equations;

This image shows the role of optimizers in updataing weights and biases, following images show how the weights are updated using momentum. If you want to use any other optimizer like ADAM, RMSprop or Adagrad you just have to update the following equations. The scripts in this repo implement differnt optimizers.

Training wiht ReLu activation

Another activation function that is still used in many popular CNN architecture is RelU, it is implemented as;

and forward and bak prop work as follows

Following images compare the results of training the network with different activations

Detailed Results

Descriptions are provided in each slide