You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If you are interested in contributing either bug fixes or new features, open an issue and we can talk about it!
5
+
New PRs will require:
6
+
7
+
1. New unit tests for any new or changed common module, and all unit tests should pass.
8
+
2. All code should follow a similar style to the rest of the repository and the linter should pass.
9
+
3. Documentation of new features.
10
+
4. Manual approval.
11
+
12
+
13
+
We use the [GitFlow](https://datasift.github.io/gitflow/IntroducingGitFlow.html) model, meaning that all PRs should be opened against the `develop` branch!
# The Autonomous Learning Library: An Object-Oriented Deep Reinforcement Learning Library in Pytorch
2
-
3
-
The Autonomous Learning Library (`all`) is an object-oriented deep reinforcement learning library in `pytorch`. The goal of the library is to provide implementations of modern reinforcement learning algorithms that reflect the way that reinforcement learning researchers think about agent design and to provide the components necessary to build and test new ideas with minimal overhead.
4
-
5
-
## Why use `all`?
6
-
7
-
The primary reason for using `all` over its many competitors is because it contains components that allow you to *build your own* reinforcement learning agents.
8
-
We provide out-of-the-box modules for:
9
-
10
-
-[x] Custom Q-Networks, V-Networks, policy networks, and feature networks
11
-
-[x] Generic function approximation
12
-
-[x] Target networks
13
-
-[x] Polyak averaging
14
-
-[x] Experience Replay
15
-
-[x] Prioritized Experience Replay
16
-
-[x] Advantage Estimation
17
-
-[x] Generalized Advantage Estimation (GAE)
18
-
-[x] Easy parameter and learning rate scheduling
19
-
-[x] An enhanced `nn` module (includes dueling layers, noisy layers, action bounds, and the coveted `nn.Flatten`)
20
-
-[x]`gym` to `pytorch` wrappers
21
-
-[x] Atari wrappers
22
-
-[x] An `Experiment` API for comparing and evaluating agents
23
-
-[x] A `SlurmExperiment` API for running massive experiments on computing clusters
24
-
-[x] A `Writer` object for easily logging information in `tensorboard`
25
-
-[x] Plotting utilities for generating paper-worthy result plots
26
-
27
-
Rather than being embedded in the agents, all of these modules are available for use by your own custom agents.
28
-
Additionally, the included agents accept custom versions of any of the above objects.
29
-
Have a new type of replay buffer in mind?
30
-
Code it up and pass it directly to our `DQN` and `DDPG` implementations.
31
-
Additionally, our agents were written with readibility as a primary concern, so they are easy to modify.
32
-
33
-
## Algorithms
1
+
# The Autonomous Learning Library: A PyTorch Library for Building Reinforcement Learning Agents
2
+
3
+
The `autonomous-learning-library` is an object-oriented deep reinforcement learning (DRL) library for PyTorch.
4
+
The goal of the library is to provide the necessary components for quickly building and evaluating novel reinforcement learning agents,
5
+
as well as providing high-quality reference implementations of modern DRL algorithms.
6
+
The full documentation can be found at the following URL: [https://autonomous-learning-library.readthedocs.io](https://autonomous-learning-library.readthedocs.io).
7
+
8
+
## Tools for Building New Agents
9
+
10
+
The primary goal of the `autonomous-learning-library` is to facilitate the rapid development of new reinforcement learning agents by providing common tools for building and evaluation agents, such as:
11
+
12
+
* A flexible function `Approximation` API that integrates features such as target networks, gradient clipping, learning rate schedules, model checkpointing, multi-headed networks, loss scaling, logging, and more.
13
+
* Various memory buffers, including prioritized experience replay (PER), generalized advantage estimation (GAE), and more.
14
+
* A `torch`-based `Environment` interface that simplies agent implementations by cutting out the `numpy` middleman.
15
+
* Common wrappers and agent enhancements for replicating standard benchmarks.
16
+
*[Slurm](https://slurm.schedmd.com/documentation.html) integration for running large-scale experiments.
17
+
* Plotting and logging utilities including `tensorboard` integration and utilities for generating common plots.
18
+
19
+
See the [documentation](https://autonomous-learning-library.readthedocs.io) guide for a full description of the functionality provided by the `autonomous-learning-library`.
20
+
Additionally, we provide an [example project](https://github.com/cpnota/all-example-project) which demonstrates the best practices for building new agents.
21
+
22
+
## High-Quality Reference Implementations
23
+
24
+
The `autonomous-learning-library` separates reinforcement learning agents into two modules: `all.agents`, which provides flexible, high-level implementations of many common algorithms which can be adapted to new problems and environments, and `all.presets` which provides specific instansiations of these agents tuned for particular sets of environments, including Atari games, classic control tasks, and PyBullet robotics simulations. Some benchmark results showing results on-par with published results can be found below:
25
+
26
+
27
+
34
28
35
29
As of today, `all` contains implementations of the following deep RL algorithms:
36
30
@@ -49,131 +43,55 @@ It also contains implementations of the following "vanilla" agents, which provid
49
43
-[x] Vanilla Q-Learning
50
44
-[x] Vanilla Sarsa
51
45
52
-
We will try to stay up-to-date with advances in the field, but we do not intend to implement every algorithm. Rather, we prefer to maintain a smaller set of high-quality agents that have achieved notoriety in the field.
53
-
54
-
We have labored to make sure that our implementations produce results comparable to published results.
55
-
Here's a sampling of performance on several Atari games:
56
-
57
-
58
-
59
-
These results were generated using the `all.presets.atari` module, the `SlurmExperiment` utility, and the `all.experiments.plots` module.
60
-
61
-
## Example
62
-
63
-
Our agents implement a single method: `action = agent.act(state, reward)`.
64
-
Much of the complexity is handled behind the scenes, making the final API simple.
65
-
Unlike many libraries, we do not combine the learning algorithm and the training loop.
66
-
Instead, our agents can be embedded in existing applications and trained in the online setting.
67
-
68
-
The `all.presets` includes agents that preconfigured for certain types of environments.
69
-
It can be used as follows:
70
-
71
-
```python
72
-
fromall.presets.atari import dqn
73
-
fromall.environments import AtariEnvironment
46
+
## Installation
74
47
75
-
env = AtariEnvironment('Breakout')
76
-
agent = dqn(lr=3e-4)(env)
48
+
First, you will need a new version of [PyTorch](https://pytorch.org) (>1.3), as well as [Tensorboard](https://pypi.org/project/tensorboard/).
49
+
Then, you can install the `autonomous-learning-library` through PyPi:
77
50
78
-
whileTrue:
79
-
if env.done:
80
-
env.reset()
81
-
else:
82
-
env.step(action)
83
-
env.render()
84
-
action = agent.act(env.state, env.reward)
85
51
```
86
-
87
-
However, generally we recommend using the `Experiment` API, which adds many additional features:
Alternately, you can install directly from this repository:
102
56
103
57
```
104
-
make tensorboard
105
-
```
106
-
107
-
## Installation
108
-
109
-
This library is built on top of `pytorch`.
110
-
If you don't want your trials to take forever, it is highly recommended that you make sure your installation has CUDA support (and that you have a CUDA compatible GPU).
111
-
You'll also need `tensorflow` in order to use `tensorboard` (used for storing and plotting runs).
112
-
113
-
There are two ways to install the `autonomous-learning-library` : a "light" installation, which assumes that the major dependencies are already installed, and a "full" installation which installs everything from scratch.
114
-
115
-
### Light Installation
116
-
117
-
Use this if you already have `pytorch` and `tensorflow` installed.
Presto! If you have any trouble with installing the Gym environments, check out their [GitHub page](https://github.com/openai/gym) and try whatever they recommend in [current year].
125
-
126
-
### Full Installation
63
+
You can also install the prerequisites using:
127
64
128
-
If you're on Linux and don't have `pytorch` or `tensorflow` installed, we did you the courtesy of providing a helpful install script:
129
-
130
-
```bash
131
-
make install
132
65
```
133
-
134
-
With any luck, the `all` library should now be installed!
135
-
136
-
### Testing Your Installation
137
-
138
-
The unit tests may be run using:
139
-
66
+
pip install autonomous-learning-library[pytorch]
140
67
```
141
-
make test
142
-
```
143
-
144
-
If the unit tests pass with no errors, it is more than likely that your installation works! The unit test run every agent using both `cpu` and `cuda` for a few timesteps/episodes.
145
68
146
69
## Running the Presets
147
70
148
-
You can easily benchmark the included algorithms using the scripts in `./benchmarks`.
149
-
To run a simple `CartPole` benchmark, run:
71
+
If you just want to test out some cool agents, the `scripts` directory contains the basic code for doing so.
150
72
151
73
```
152
-
python scripts/classic.py CartPole-v1 dqn
74
+
python scripts/atari.py Breakout a2c
153
75
```
154
76
155
-
Results are printed to the console, and can also be viewed by running:
77
+
You can watch the training progress using:
156
78
157
79
```
158
-
make tensorboard
80
+
tensorboard --logdir runs
159
81
```
160
82
161
83
and opening your browser to http://localhost:6006.
162
-
163
-
To run an Atari benchmark in CUDA mode (warning: this could take several hours to run, depending on your machine):
84
+
Once the model is trained to your satisfaction, you can watch the trained model play using:
If you want to run in `cpu` mode (~10x slower on my machine), you can add ```--device cpu```:
170
-
171
-
```
172
-
python scipts/atari.py Pong dqn --device cpu
173
-
```
90
+
where `id` is the ID of your particular run. You should should be able to find it using tab completion or by looking in the `runs` directory.
91
+
The `autonomous-learning-library` also contains presets and scripts for classic control and PyBullet environments.
174
92
175
93
## Note
176
94
177
-
This library was built at the [Autonomous Learning Laboratory](http://all.cs.umass.edu) (ALL) at the [University of Massachusetts, Amherst](https://www.umass.edu).
95
+
This library was built in the [Autonomous Learning Laboratory](http://all.cs.umass.edu) (ALL) at the [University of Massachusetts, Amherst](https://www.umass.edu).
178
96
It was written and is currently maintained by Chris Nota (@cpnota).
179
97
The views expressed or implied in this repository do not necessarily reflect the views of the ALL.
0 commit comments