This diagram shows the structure of a feedforward Multilayer Perceptron (MLP).
The network consists of an input layer, multiple hidden layers, and an output layer.
Each neuron is fully connected to the next layer, and data flows forward through the network.
(Note: This is a simplified illustration, not a real computation.)
Let’s follow a single number as it flows through the network:
Input: x = 1
↓
Layer 1: Multiply → x = 1 × 25 = 25 (Weight)
↓
Layer 2: ReLU → x = max(0, 25) = 25 (Activation)
↓
Output: x = 25
Target value: 30
Loss: Error = Predicted - Target = 25 - 30 = -5
Now let’s simulate how the model would learn by adjusting the weight.
Let’s go back to Layer 1 and increase the weight from 25 to 26 (to reduce the error and get closer to the target).
New weight = 26 (was 25 before)
Input: x = 1
↓
Layer 1: Multiply → x = 1 × 26 = 26
↓
Layer 2: ReLU → x = max(0, 26) = 26
↓
Output: x = 26
Target value: 30
Loss: Error = Predicted - Target = 26 - 30 = -4
The model learned! The prediction got closer to the correct value, and the error went down from –5 to –4.
In this example, we use a very simple loss function:
Loss = (Prediction - Target)²
This helps the model understand how wrong it is, so it can adjust its weights during learning.
Example:
Prediction = 25
Target = 30
Loss = (25 - 30)² = 25
Gradient descent is the process of adjusting the weights in small steps to reduce the loss.
In our example, we made a small change to the weight (from 25 to 26),
which brought the prediction closer to the target value.
This is the core idea of how neural networks learn.
Overfitting happens when the model becomes too good at memorizing the training data,
but performs poorly on new, unseen data.
This can happen when:
- the model is too big (too many neurons/layers),
- training goes on for too long,
- there's no validation set to check performance,
- or there's no regularization to keep the model simple.
In short: bigger isn't always better especially with limited data.
def feedforward(X, weights, biases):
activations = [X]
pre_activations = []
for i in range(len(weights) - 1):
Z = np.dot(activations[-1], weights[i]) + biases[i]
pre_activations.append(Z)
A = relu(Z)
activations.append(A)
Z = np.dot(activations[-1], weights[-1]) + biases[-1]
pre_activations.append(Z)
A = softmax(Z)
activations.append(A)
return activations, pre_activations-
The function starts by taking the input values.
-
We go through all layers except the last one and do the math: input × weight + bias.
-
Then we apply the ReLU activation (like a filter).
-
We save the results and continue to the next layer.
-
In the last layer, we do the same, but use softmax instead of ReLU.
-
At the end, we return all the values from each layer (before and after activation).
def backprop(activations, pre_activations, y_true, weights):
m = y_true.shape[0]
grads_W, grads_b = [], []
dZ = activations[-1] - y_true
for i in reversed(range(len(weights))):
dW = np.dot(activations[i].T, dZ) / m
db = np.sum(dZ, axis=0, keepdims=True) / m
grads_W.insert(0, dW)
grads_b.insert(0, db)
if i > 0:
dA = np.dot(dZ, weights[i].T)
dZ = dA * relu_derivative(pre_activations[i - 1])
return grads_W, grads_b-
We start from the output error: predicted − true (dZ = activations[-1] - y_true).
-
m is the batch size (how many samples are in this pass).
-
Then we go layer by layer backwards through the network.
-
For each layer we compute how much the weights and biases should change (dW, db) and store them.
-
If there is a previous layer, we push the error back through the weights, and apply the ReLU derivative to keep only the parts that were active.
-
At the end we return all gradients for weights and biases (these will be used to update the parameters).