A comprehensive reinforcement learning project implementing various RL algorithms on a customizable GridWorld environment. This project features multiple agents including Q-Learning, Deep Q-Learning, Policy Iteration, and Value Iteration algorithms.
- Python 3.8 or higher
- pip (Python package installer)
- Clone the repository (if you haven't already):
git clone https://github.com/Amine-Amllal/GridWord.git
cd GridWord- Create a virtual environment:
On Windows (PowerShell):
python -m venv RL
.\RL\Scripts\Activate.ps1On Windows (Command Prompt):
python -m venv RL
RL\Scripts\activate.batOn macOS/Linux:
python3 -m venv RL
source RL/bin/activate- Install required packages:
pip install -r requirements.txt- Verify installation:
python -c "import numpy, matplotlib, imageio; print('All dependencies installed successfully!')"Once the environment is set up, you can run any of the agents from their respective directories:
# Q-Learning Agent
python Q-Learning/q_learning_agent.py
# Dynamic Q-Learning Agent (with moving obstacles)
python Q-Learning/q_learning_agent_dynamic.py
# Q-Learning with Reward Shaping
python Q-Learning/q_learning_avec_shaping.py
# Deep Q-Learning Agent (Neural Network)
python NeuralQ-Learning/neural_network_q_value_agent.py
# DQN DeepMind Implementation
python DQN_DeepMind/dqn_agent.py
# DQN with Moving Goal
python DQN_DeepMind_MovingGoal/dqn_agent_moving_goal.py
# PPO Agent (Stable-Baselines3)
python StableBase3Zoo_PPO/run_simulation.py
# Policy Iteration Agent
python PolicyIteration/policy_iteration_agent.py
# Value Iteration Agent
python ValueIteration/value_iteration_agent.py
# Random Agent (baseline)
python RandomAgent/random_agent.pyThe GridWorld environment (frozenlake_env.py) provides a highly customizable frozen lake simulation where:
- Grid Size: Users can define custom grid dimensions (rows × columns)
- Start Position: Configurable starting position for the agent
- Goal Position: Users can place the goal anywhere on the grid
- Obstacles (Holes): Fully customizable obstacle positions
- Dynamic Elements: Support for moving goals and obstacles (in dynamic agent mode)
from frozenlake_env import make_frozen_lake
# Create a custom 8x8 environment
env = make_frozen_lake(
nrow=8,
ncol=8,
start_state=(0, 0),
goal=(7, 7),
holes=[(2, 2), (3, 5), (5, 3), (6, 6)]
)- S (Start): The agent's starting position
- F (Frozen): Safe cells where the agent can walk
- H (Hole): Obstacles that terminate the episode with negative reward
- G (Goal): Target destination with positive reward
The agent can perform 4 actions: LEFT, DOWN, RIGHT, UP
The Q-Learning agent implements the classic tabular Q-Learning algorithm, learning optimal policies through trial and error. This agent maintains a Q-table that stores action values for each state-action pair.
- Epsilon-greedy exploration strategy
- Adaptive learning with decay
- Visualization of learned policy
- Path animation showing agent navigation
Cumulative Rewards During Training:
This graph shows how the agent's performance improves over episodes, with cumulative rewards increasing as the agent learns better policies.
Final Learned Path:
The agent successfully navigates from the start position to the goal while avoiding obstacles, following the optimal policy learned through Q-Learning.
Training Animation:
This animation visualizes the learning process, showing how the agent explores the environment and gradually improves its path-finding strategy.
- Learns optimal policy within hundreds of episodes
- Achieves high success rate in reaching the goal
- Adapts to different environment configurations
The Deep Q-Learning agent uses a neural network to approximate Q-values, enabling it to generalize across states and handle more complex environments. This implementation uses a custom feedforward neural network built with NumPy.
- Multi-layer neural network architecture
- Experience-based learning
- State encoding for neural input
- Gradient descent optimization
- Policy visualization with heatmaps
- Input Layer: State encoding (position + grid features)
- Hidden Layers: Configurable depth and width
- Output Layer: Q-values for each action
- Activation: ReLU for hidden layers
Learned Policy Visualization:
This heatmap shows the learned policy across the grid, with arrows indicating the optimal action for each state. The color intensity represents the maximum Q-value at each position.
Training Progress:
The training progress charts display:
- Episode rewards over time
- Success rate improvement
- Loss convergence
- Average Q-values evolution
- Better generalization than tabular Q-Learning
- Smoother learning curves with experience replay
- Capable of handling larger state spaces
- More stable convergence with proper hyperparameter tuning
A professional implementation of Deep Q-Network following DeepMind's architecture:
- Experience replay buffer
- Target network for stability
- GIF generation capabilities
- Comprehensive training metrics
See DQN_DeepMind/README.md for detailed documentation.
An advanced DQN implementation that handles dynamic environments:
- Randomized goal positions during training
- Optimized replay memory strategies
- Goal distribution verification
- Memory type comparison tools
See DQN_DeepMind_MovingGoal/README.md for detailed documentation.
Proximal Policy Optimization implementation using the Stable-Baselines3 library:
- State-of-the-art policy gradient method
- Visual demonstrations of learned behavior
- Easy-to-use interface for RL experiments
See StableBase3Zoo_PPO/README.md for detailed documentation.
Implements dynamic programming to compute the optimal policy by iteratively improving the policy until convergence.
Uses dynamic programming to compute optimal value function and derives the optimal policy from it.
Baseline agent that takes random actions, useful for comparison with learning agents.
GridWord/
├── frozenlake_env.py # Customizable environment
├── requirements.txt # Python dependencies
├── README.md # This file
├── LICENSE # License file
├── RL/ # Virtual environment
├── Q-Learning/ # Q-Learning implementations
│ ├── q_learning_agent.py # Tabular Q-Learning
│ ├── q_learning_agent_dynamic.py # Q-Learning with dynamic obstacles
│ └── q_learning_avec_shaping.py # Q-Learning with reward shaping
├── NeuralQ-Learning/ # Neural network Q-Learning
│ ├── neural_network_q_value_agent.py # Deep Q-Learning implementation
│ ├── neural_q_learning_policy.png
│ └── neural_q_learning_training_progress.png
├── DQN_DeepMind/ # DeepMind-style DQN implementation
│ ├── dqn_agent.py # Main DQN agent
│ ├── create_gif_from_trained_agent.py
│ ├── generate_agent_gif.py
│ ├── test_gif_generation.py
│ ├── README.md
│ ├── requirements.txt
│ └── dqn_results/ # Training results
├── DQN_DeepMind_MovingGoal/ # DQN with dynamic goal
│ ├── dqn_agent_moving_goal.py # DQN for moving goal environments
│ ├── compare_memory_types.py # Replay memory comparison
│ ├── verify_goal_distribution.py
│ ├── README.md
│ ├── requirements.txt
│ └── dqn_moving_goal_results_*/ # Training results
├── StableBase3Zoo_PPO/ # PPO implementation using Stable-Baselines3
│ ├── run_simulation.py # PPO agent runner
│ ├── README.md
│ ├── requirements.txt
│ └── ppo_visual_demo_*/ # Training visualizations
├── PolicyIteration/ # Policy Iteration algorithm
│ └── policy_iteration_agent.py
├── ValueIteration/ # Value Iteration algorithm
│ └── value_iteration_agent.py
├── RandomAgent/ # Random baseline agent
│ └── random_agent.py
└── q_learning_results_*/ # Legacy Q-Learning results
Each agent automatically saves results to timestamped folders containing:
- Training animations (GIF)
- Policy visualizations (PNG)
- Performance metrics (PNG/TXT)
- Q-tables or network parameters (CSV/NPY)
See the LICENSE file for details.
Contributions are welcome! Feel free to:
- Add new RL algorithms
- Improve environment features
- Enhance visualizations
- Optimize training performance
For questions or suggestions, please open an issue on GitHub.
Happy Learning! 🚀