A from-scratch implementation of two foundational supervised classifiers in PyTorch — logistic regression and a multi-layer fully-connected neural network (MLP) — built by defining the forward pass, layer composition, and training loop directly rather than relying on high-level abstractions. The goal is to make every moving part of a feed-forward classifier explicit: linear layers, non-linear activations, the loss surface, and gradient-based optimization.
Both models are trained end-to-end on a multi-class image-classification task. The project builds up from the simplest possible linear classifier to a deeper non-linear network, so the effect of depth, activation functions, and regularization can be observed directly.
- Logistic regression — a single linear layer followed by a softmax, trained with cross-entropy loss. Serves as the linear baseline.
- Fully-connected network (MLP) — a configurable stack of linear layers with non-linear activations (sigmoid / tanh / ReLU), demonstrating how added depth and non-linearity lift accuracy above the linear baseline.
- Modular layer construction using a
ModuleDict-based design, so the network architecture (number of layers, hidden widths, activation choice) is data-driven and easy to reconfigure. - Forward propagation for linear layers and element-wise non-linearities, with PyTorch autograd handling the backward pass.
- Training loop with mini-batch SGD, configurable learning rate, weight decay (L2 regularization), momentum, and epoch/step scheduling.
- Parameter initialization and a clean separation between model definition and training policy.
- Train/validation evaluation with accuracy tracking across epochs to diagnose under- vs. over-fitting.
- How a softmax + cross-entropy classifier is the multi-class generalization of logistic regression.
- Why non-linear activations are what let an MLP represent decision boundaries a linear model cannot.
- The practical mechanics of gradient-based training: learning-rate choice, weight decay, and the bias–variance trade-off visible in train-vs-validation curves.
.
├── logistic_regression.py # linear softmax classifier
├── fully_connected.py # configurable MLP (ModuleDict-based layers)
├── train.py # mini-batch training loop + evaluation
├── create_dataset.py # data loading / preprocessing
└── README.md
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install torch numpy# Train the logistic-regression baseline
python train.py --model logistic_regression
# Train the fully-connected network
python train.py --model fully_connectedPython · PyTorch · NumPy
This is a pedagogical, from-scratch implementation — the emphasis is on clarity and correctness of the underlying mechanics rather than on squeezing out state-of-the-art accuracy. It serves as the foundation for the convolutional and recurrent architectures explored in companion projects.