This project implements and compares three Reinforcement Learning algorithms using Linear Value Function Approximation (Tile Coding) on the LunarLander-v3 environment:
- n-step SARSA Semi-Gradient
- SARSA(lambda) Semi-Gradient
- REINFORCE with Baseline Semi-Gradient
The code generates an organized directory structure for outputs used for performance analysis and reporting.
Best Weights_{Algorithm}_{Timestamp}.npy: The saved weight matrix for the best-performing policy discovered during tuning.
Raw Results_{Algorithm}_{Timestamp}.csv: The complete raw dataset from the parallelized LHD search.
Learning Curve_{Algorithm}_{Timestamp}.csv: Learning curve data used for aggregate plotting../Results/Top 10/Top 10_{Algorithm}_{Timestamp}.csv: Ranked configurations based on the Algorithm Score../Results/Top 5/Top 5_{Algorithm}_{Timestamp}.csv: Absolute champions ranked by SP95LB../Results/Executive_Summary_Results.csv: Aggregated performance of the superlative policies across all algorithms../Results/Superlative Hyperparameters/Superlative Hyperparameters_{Algorithm}_{Timestamp}.txt: Record of tuned hyperparameters../Results/Superlative Hyperparameters/Analysis_{Algorithm}_{Timestamp}.txt: Response Surface Methodology (RSM) regression and ANOVA output../Results/Training Plots/Training Plot_{Algorithm}_{Timestamp}.png: Standard learning curves generated during training../Results/LHD Plots/LHD Plot_{Algorithm}_{Timestamp}.png: Learning curves specifically for the superlative configurations.Combined_Superlative_Learning_Curves.png: Comparative plot of all superlative algorithm configurations../Results/Superlative Video/: MP4 recordings of the agent playing in the environment using the superlative weights.
Install the required dependencies using the requirements.txt file:
pip install -r requirements.txtExecute the Semi Grad SARSA script. This trains the agent, performs a Latin Hypercube Design (LHD) search for hyperparameters, and saves the superlative results.
python3 Semi_Grad_SARSA.pyExecute the Lambda SARSA script to perform the same kind of tuning and training process for step 1.
python3 Lambda_SARSA.pyExecute the REINFORCE script to perform the same kind of tuning and training process for steps 1, 2. '''bash python3 REINFORCE.py '''
Once all three algorithms have been run and their respective LC_*.csv files exist in the Results folder, generate the combined comparison graph:
python3 Combined_Learning_Curves.pyExecute superlative video generation to show the best policies in action.
python3 Superlative_Video.py
'''
### Bash Command to run all files in order
```bash
python3 Semi_Grad_SARSA.py && python3 Lambda_SARSA.py && python3 REINFORCE.py && python3 Combined_Learning_Curves.py && python3 Superlative_Video.py
'''
---
### Configuration
Hyperparameters (such as learning rate decay, $n$-steps, or $\lambda$), training iterations ($M$), and statistical replications ($Z$) can be modified within the `HP` dataclass located at the top of `Semi_Grad_SARSA.py`, `Lambda_SARSA.py`, and `REINFORCE.py`.
To visually see the agent play after training, uncomment the `display_best_policy` call in the `__main__` block of the algorithm scripts.
## Project Structure & File Descriptions
* **`Semi_Grad_SARSA.py`**: Implements the n-step SARSA Semi-Gradient algorithm, including bootstrapping logic and hyperparameter definitions.
* **`Lambda_SARSA.py`**: Implements the SARSA(lambda) Semi-Gradient algorithm using sparse accumulating eligibility traces.
* **`REINFORCE.py`**: Implements the REINFORCE with Baseline algorithm with numeric stability and backward return calculation.
* **`utils.py`**: The project's core engine; contains the Tile Coding implementation, experimental pipelines, LHD sampling logic, and statistical analysis functions.
* **`Algorithm Pseudo Code.txt`**: Contains the pseudocode for the implemented algorithms (n-step SARSA, SARSA(lambda) and Reinforce with baseline).
* **`Combined_Learning_Curves.py`**: Utility script to aggregate results and generate comparative superlative learning curves.
* **`Superlative_Video.py`**: Script to record and save superlative policy performance as MP4 videos.
* **`requirements.txt`**: Lists necessary libraries including `gymnasium[lunar-lander]`, `statsmodels`, `scipy`, and `joblib`.
* **`Results/`**: (Folder) Contains sub-directories for plots, CSV metrics, and analysis files.
* **`Saved Models/`**: (Folder) Contains the saved `.npy` weight matrices for the best-performing policies.
* **`Raw Results/`**: (Folder) Contains the full CSV datasets from LHD searches.
---
## Runtime Optimizations
Several optimizations have been implemented to ensure the experiments remain tractable while maintaining algorithmic integrity:
1. **Sparse Eligibility Traces (SARSA($\lambda$))**: Instead of performing dense matrix operations over the entire Index Hash Table ($2^{18}$ entries) every timestep, the implementation tracks only the indices of non-zero traces. This reduces the complexity of the trace update from $O(\text{IHT Size})$ to $O(\text{Active Traces})$. This optimization is **functionally identical** to the standard algorithm but provides a massive speedup.
2. **Parallel Hyperparameter Search**: The Latin Hypercube Design (LHD) experiment in `utils.py` uses `joblib` to distribute different hyperparameter configurations across multiple CPU cores. Since each experimental run is independent, this **does not change** the results but allows for significantly more exhaustive tuning within the same timeframe.
3. **Vectorized Weight Updates**: Across all algorithms, weight updates and Q-value calculations use NumPy's vectorized operations (e.g., `w[active_features, action].sum()`). This leverages optimized C-level implementations for linear algebra, ensuring the linear value function approximation is calculated efficiently without changing the underlying math.
4. **Numerical Stability in Softmax**: In the REINFORCE implementation, the preference calculation subtracts the maximum preference value before exponentiation. This prevents floating-point overflow/underflow without altering the resulting probability distribution.