Multilayer Perceptron

This project is about building a neural network from scratch without using any deep learning framework such as TensorFlow or PyTorch.
The goal is to understand and manually implement the core mathematical foundations behind neural networks, including: Feedforward, Backpropagation, Gradient Descent,
as well as key components like
ReLU, ReLU Derivative, Softmax, and the Loss Function.

All of these are implemented from scratch, providing a complete understanding of how a neural network learns.

---

Introduction

The project implements a Multilayer Perceptron (MLP) to predict whether a breast cancer case is malignant (M) or benign (B)
based on the Wisconsin Diagnostic Breast Cancer dataset.
This is a binary classification task.

---

Objectives

• Understand the architecture and logic behind neural networks.
• Implement training manually using
  Feedforward, Backpropagation, and Gradient Descent.
• Visualize model performance using loss and accuracy curves.
• Learn how activation functions and derivatives influence training.

---

Feedforward

Feedforward is the process of passing inputs through the network layer by layer to obtain an output prediction.
Each layer computes a weighted sum of its inputs, adds a bias, and applies an activation function.

Z = W · X + b
A = f(Z)

Where:

• ( W ): weights
• ( X ): inputs
• ( b ): biases
• ( f ): activation function
• ( A ): activated output

This step produces the model’s predicted output.

---

Backpropagation

Backpropagation computes how much each weight contributed to the final error.
It uses the chain rule from calculus to propagate the gradient backward through the network.

Steps:

1. Compute the loss (difference between prediction and target).
2. Calculate partial derivatives of the loss with respect to each weight.
3. Propagate the error from the output layer back to the input layer.

This allows the network to learn which weights need to increase or decrease to minimize the loss.

---

Gradient Descent

Gradient Descent updates the weights to minimize the loss function.
It moves in the opposite direction of the gradient.

W := W - η * ∂L/∂W

Where:

• η — learning rate
• L — loss function

This process repeats for many epochs until the model converges to a minimum loss.

---

Activation Functions

Activation functions introduce non-linearity, enabling the network to learn complex relationships in data.

---

ReLU (Rectified Linear Unit)

The ReLU activation function replaces all negative values with zero
and keeps positive values unchanged.
It introduces non-linearity, which helps the model learn complex patterns.

Mathematical definition:

ReLU(x) = max(0, x)

Example:

Input:  [ 5, -3, 0, 8, -1, 2 ]
Output: [ 5,  0, 0, 8,  0, 2 ]

---

ReLU Derivative

The derivative of ReLU is used during backpropagation to compute gradients efficiently. It is 1 for positive values and 0 for negative or zero inputs.

Example:

Input:    [ 5, -3, 0, 8, -1, 2 ]
ReLU'(x): [ 1,  0, 0, 1,  0, 1 ]

---

Softmax

The Softmax activation function converts raw scores (logits) into a probability distribution, where the sum of all outputs equals 1. It is commonly used in the output layer for classification tasks.

Example:

Input:  [2.0, 1.0, 0.1]
Output: [0.659, 0.242, 0.099]
Sum:    1.000

---

Loss Function

The project uses Binary Cross-Entropy (BCE) loss to measure the difference between the predicted probability and the true label. This function penalizes incorrect predictions more strongly the more confident they are.

L = -(1/N) * Σ [ yᵢ log(pᵢ) + (1 - yᵢ) log(1 - pᵢ) ]

---

Training Overview

1. Initialize weights and biases randomly (He/Xavier initialization).

2. Feedforward: compute activations layer by layer.

3. Compute Loss: measure the prediction error.

4. Backpropagate: calculate gradients.

5. Gradient Descent: update weights and biases.

6. Repeat: for each epoch until loss converges.

During training, the program displays both training loss and validation loss, and generates learning curves to visualize model performance.

---

Prediction Phase

After training, the model:

1. Loads the saved weights and biases.

2. Performs feedforward on unseen (validation/test) data.

3. Outputs a prediction (M or B) with probability.

4. Evaluates performance using Binary Cross-Entropy and accuracy metrics.

---

Bonus Ideas

• Early stopping — training stops when the validation loss stops improving (with patience).
• Metrics history — training/validation loss and accuracy are saved to training_logs/metrics.json.
• Multiple metrics — both loss and accuracy are computed and logged per epoch and plotted. For each layer the project exports:
• Weight heatmaps → model_visualizations/weights_layer_{i}.png
• Bias histograms → model_visualizations/biases_layer_{i}.png
• Compare learning curves — compare_runs.py plots multiple runs on one chart.
• Alternative optimizer — e.g., Adam/RMSprop/Nesterov (selectable via a flag).