Switch to Markdown documentation (MyST parser) (#2219)

* Switch to mardown documentation (MyST parser)

* Update requirements

* Doc fix following copilot suggestions

Author: araffin

.github/PULL_REQUEST_TEMPLATE.md

@@ -20,7 +20,7 @@
 <!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
 <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
 - [ ] I've read the [CONTRIBUTION](https://github.com/DLR-RM/stable-baselines3/blob/master/CONTRIBUTING.md) guide (**required**)
-- [ ] I have updated the changelog accordingly (**required**).
+- [ ] I have updated the changelog accordingly (`docs/misc/changelog.md`) (**required**).
 - [ ] My change requires a change to the documentation.
 - [ ] I have updated the tests accordingly (*required for a bug fix or a new feature*).
 - [ ] I have updated the documentation accordingly.

CONTRIBUTING.md

@@ -112,7 +112,7 @@ make spelling
 
 ## Changelog and Documentation
 
-Please do not forget to update the changelog (`docs/misc/changelog.rst`) and add documentation if needed.
+Please do not forget to update the changelog (`docs/misc/changelog.md`) and add documentation if needed.
 You should add your username next to each changelog entry that you added. If this is your first contribution, please add your username at the bottom too.
 A README is present in the `docs/` folder for instructions on how to build the documentation.
 

docs/README.md

@@ -1,4 +1,4 @@
-## Stable Baselines3 Documentation
+# Stable Baselines3 Documentation
 
 This folder contains documentation for the RL baselines.
 

docs/common/atari_wrappers.rst docs/common/atari_wrappers.mddocs/common/atari_wrappers.rst renamed to docs/common/atari_wrappers.md

@@ -1,7 +1,8 @@
-.. _atari_wrapper:
+(atari-wrapper)=
 
-Atari Wrappers
-==============
+# Atari Wrappers
 
+```{eval-rst}
 .. automodule:: stable_baselines3.common.atari_wrappers
   :members:
+```

docs/common/distributions.md

@@ -0,0 +1,28 @@
+(distributions)=
+
+# Probability Distributions
+
+Probability distributions used for the different action spaces:
+
+- `CategoricalDistribution` -> Discrete
+- `DiagGaussianDistribution` -> Box (continuous actions)
+- `StateDependentNoiseDistribution` -> Box (continuous actions) when `use_sde=True`
+
+% - ``MultiCategoricalDistribution`` -> MultiDiscrete
+
+% - ``BernoulliDistribution`` -> MultiBinary
+
+The policy networks output parameters for the distributions (named `flat` in the methods).
+Actions are then sampled from those distributions.
+
+For instance, in the case of discrete actions. The policy network outputs probability
+of taking each action. The `CategoricalDistribution` allows sampling from it,
+computes the entropy, the log probability (`log_prob`) and backpropagate the gradient.
+
+In the case of continuous actions, a Gaussian distribution is used. The policy network outputs
+mean and (log) std of the distribution (assumed to be a `DiagGaussianDistribution`).
+
+```{eval-rst}
+.. automodule:: stable_baselines3.common.distributions
+  :members:
+```

docs/common/distributions.rst

@@ -1,7 +1,8 @@
-.. _env_checker:
+(env-checker)=
 
-Gym Environment Checker
-========================
+# Gym Environment Checker
 
+```{eval-rst}
 .. automodule:: stable_baselines3.common.env_checker
   :members:
+```

docs/common/env_checker.rst docs/common/env_checker.mddocs/common/env_checker.rst renamed to docs/common/env_checker.md

@@ -1,7 +1,8 @@
-.. _env_util:
+(env-util)=
 
-Environments Utils
-=========================
+# Environments Utils
 
+```{eval-rst}
 .. automodule:: stable_baselines3.common.env_util
   :members:
+```

docs/common/env_util.rst docs/common/env_util.mddocs/common/env_util.rst renamed to docs/common/env_util.md

@@ -1,24 +1,26 @@
-.. _envs:
+(envs)=
 
+```{eval-rst}
 .. automodule:: stable_baselines3.common.envs
 
 
+```
 
-Custom Environments
-===================
+# Custom Environments
 
 Those environments were created for testing purposes.
 
+## BitFlippingEnv
 
-BitFlippingEnv
---------------
-
+```{eval-rst}
 .. autoclass:: BitFlippingEnv
   :members:
 
+```
 
-SimpleMultiObsEnv
------------------
+## SimpleMultiObsEnv
 
+```{eval-rst}
 .. autoclass:: SimpleMultiObsEnv
-  :members:
+  :members:
+```

docs/common/envs.rst docs/common/envs.mddocs/common/envs.rst renamed to docs/common/envs.md

@@ -1,7 +1,8 @@
-.. _eval:
+(eval)=
 
-Evaluation Helper
-=================
+# Evaluation Helper
 
+```{eval-rst}
 .. automodule:: stable_baselines3.common.evaluation
   :members:
+```