02. PyTorch Neural Network Classification - sigmoid not working for the problem?

[cleverll]

Using ReLU as activation function works for the problem, if I replace it with sigmoid, the model doesn't seem to be able to learn. I am using the exact same layers and number of neurons. The class is defined as the following:

    def __init__(self):
      super().__init__()
      self.layer_1 = nn.Linear(in_features=2, out_features=5, bias=True) 
      self.layer_2 = nn.Linear(in_features=5, out_features=5, bias=True)
      self.layer_3 = nn.Linear(in_features=5, out_features=1) 
      self.sigmoid = nn.Sigmoid()

    def forward(self, x) -> torch.Tensor:
      # no need to put activation function because the logit is passed to a sigmoid function later on
      return self.layer_3(self.sigmoid(self.layer_2(self.sigmoid(self.layer_1(x)))))

Sigmoid function should be sufficient to train the model, as I tested it on the tensorflow playground website. tAnyone knows why it's not working?

[de-lazurenko]

The issue you're experiencing is a classic case of the vanishing gradient problem.
When you use sigmoid as a hidden layer activation, it squashes all values into a narrow range (0, 1). During backpropagation, gradients are multiplied at each layer - and since sigmoid derivatives max out at 0.25, after just 2–3 layers those gradients become extremely small. The weights barely update, and the model appears stuck.
ReLU avoids this because its gradient is simply 1 for any positive input - it passes the signal through without squashing it.

Why did it work on TensorFlow Playground?
The Playground uses very clean, well-scaled 2D data and a relatively forgiving optimizer setup. It masks the vanishing gradient issue. Real training pipelines are less forgiving.

What you can do to make sigmoid work:

1. Normalize your input data - sigmoid is much more sensitive to input scale than ReLU. Make sure your features are scaled (e.g. StandardScaler or min-max normalization).
2. Adjust weight initialization - PyTorch's default kaiming_uniform_ init is designed for ReLU. For sigmoid, Xavier/Glorot initialization tends to work better:

nn.init.xavier_uniform_(self.layer_1.weight)
nn.init.xavier_uniform_(self.layer_2.weight)

3. Lower your learning rate - sigmoid networks are more sensitive to it.
   That said, for most practical cases ReLU (or its variants like LeakyReLU) is simply a better default for hidden layers.

[cleverll]

That makes a lot of sense. Thank you very much for the detailed explanation.