This project builds a PyTorch reinforcement learning library from the mathematics outward. Algorithms are developed as a complete path from the objective and estimator to the data flow and parameter update.
The implemented on-policy stack runs from one-step actor-critic through PPO. DQN and Double DQN are the current implementation focus. The longer direction includes continuous and offline RL, imitation and model-based learning, Gymnasium and MuJoCo benchmarks, and robotics policy deployment.
Active development: Public APIs may change while shared training components and benchmark interfaces are stabilized.
- One-step actor-critic (Actor-Critic, baseline)
- A2C (GAE Actor-Critic)
- Natural Policy Gradient (derivation)
- TRPO (derivation)
- PPO-Clip (derivation in progress)
- DQN: derivation; implementation in progress
- Double DQN: derivation and implementation in progress
- DDPG: Deterministic Policy Gradient and DDPG derivations complete
The derivations are working specifications for the code. They determine what the buffers preserve, where gradients stop, which quantities remain fixed during an update, and how training is divided into reusable components. This is visible in boundary-correct GAE, old-policy handling in PPO, matrix-free Fisher products, and TRPO's parameter rollback.
DQN is the current off-policy implementation target. Its derivation separates generalized Q-learning into independently scheduled data collection, target-network refresh, and Q-function fitting. That decomposition defines the off-policy system now being built.
Browse all mathematical derivations
Python 3.12 or later is required. Dependencies are managed with uv.
git clone https://github.com/SaiSampathKedari/Deep-Reinforcement-Learning.git
cd Deep-Reinforcement-Learning
uv syncNow: DQN, Double DQN, and the shared replay and Q-learning system.
Next: Reusable policy, value, and Q-function interfaces; configurable optimizers; DDPG, TD3, SAC; and multi-seed Gymnasium and MuJoCo benchmarks.
Later: Offline RL, imitation learning, model-based RL, richer policy architectures, real-robot deployment, and VLA post-training.
- Mathematical foundations: Real Analysis | Probability and Distribution Theory | Statistical Inference Theory
- Sequential-decision theory: Sequential Decision Making
- Classical reinforcement learning: Reinforcement Learning
- Email: sampath@umich.edu
- LinkedIn: Sai Sampath Kedari
Released under the MIT License.