This repository implements two fundamental sequence models from scratch using NumPy: Recurrent Neural Networks (RNNs) and Long Short-Term Memory Networks (LSTMs). These models are trained for character-level language modeling, offering insight into how sequential neural networks function internally—without deep learning libraries.
A Recurrent Neural Network (RNN) processes sequential data by maintaining a hidden state vector that passes information from one timestep to the next. This mechanism enables the model to learn patterns across time or sequence.
Mathematical Formulation:
RNNs are trained using Backpropagation Through Time (BPTT), but they are limited by vanishing/exploding gradients, making it difficult to learn long-term dependencies.
Long Short-Term Memory (LSTM) networks address the limitations of vanilla RNNs by introducing gates that control the flow of information and help preserve gradients.
Main Equations:
Where the gates ((i, f, o)) regulate input, memory retention, and output—enabling the network to model longer-term dependencies.
Model Implementations
rnn.py: Contains a class for the vanilla character-level RNN, with manual forward and backward (BPTT) passes.lstm_scratch.py: Implements an LSTM from scratch, including cell/gate logic and memory updates.
Training Scripts
training2.py: Trains the RNN model, fetches data, preprocesses it, starts training, and prints sampled text after epochs.training_loop.py: Hosts an analogous training loop for the LSTM model.