Save and restore the critic's hidden state in compute_returns()#206
Merged
ClemensSchwarke merged 1 commit intoMay 4, 2026
Merged
Conversation
epalmaEth
approved these changes
May 4, 2026
jnskkmhr
added a commit
to jnskkmhr/rsl_rl
that referenced
this pull request
Jun 24, 2026
* Add Torch compile for PPO and Distillation (leggedrobotics#199) --------- Co-authored-by: adenzler-nvidia <adenzler@nvidia.com> * Clean up PPO by moving extension logic to the extension files (leggedrobotics#200) * Bump version to 5.1.0 * Add clip and constant std functionalities to Gaussian Distribution (leggedrobotics#201) --------- Co-authored-by: epalmaEth <epalma@student.ethz.ch> Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * Logger: union keys across the window instead of sampling from index 0 (leggedrobotics#205) --------- Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * Save and restore the critic's hidden state in compute_returns (leggedrobotics#206) * Adds a Beta distribution (leggedrobotics#202) * Bump version to 5.3.0 * add numpy to mock imports for building the docs * Enable custom logging services (leggedrobotics#211) * Preserve observation tensors dtype in the rollout storage (leggedrobotics#212) --------- Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * fix: include day count in long training log durations (leggedrobotics#210) --------- Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * Fix the CNNModel to support 2D observations only (leggedrobotics#215) * Bump version to 5.4.1 --------- Co-authored-by: Clemens Schwarke <96480707+ClemensSchwarke@users.noreply.github.com> Co-authored-by: adenzler-nvidia <adenzler@nvidia.com> Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> Co-authored-by: Emilio Palma <115191961+epalmaEth@users.noreply.github.com> Co-authored-by: epalmaEth <epalma@student.ethz.ch> Co-authored-by: Kevin Zakka <kevinzakka@users.noreply.github.com> Co-authored-by: Julia Beliaeva <juliabeliaeva@users.noreply.github.com> Co-authored-by: Masahiro Kubota <42679530+masahiro-kubota@users.noreply.github.com>
jnskkmhr
added a commit
to jnskkmhr/rsl_rl
that referenced
this pull request
Jun 24, 2026
* Add Torch compile for PPO and Distillation (leggedrobotics#199) --------- Co-authored-by: adenzler-nvidia <adenzler@nvidia.com> * Clean up PPO by moving extension logic to the extension files (leggedrobotics#200) * Bump version to 5.1.0 * Add clip and constant std functionalities to Gaussian Distribution (leggedrobotics#201) --------- Co-authored-by: epalmaEth <epalma@student.ethz.ch> Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * Logger: union keys across the window instead of sampling from index 0 (leggedrobotics#205) --------- Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * Save and restore the critic's hidden state in compute_returns (leggedrobotics#206) * Adds a Beta distribution (leggedrobotics#202) * Bump version to 5.3.0 * add numpy to mock imports for building the docs * Enable custom logging services (leggedrobotics#211) * Preserve observation tensors dtype in the rollout storage (leggedrobotics#212) --------- Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * fix: include day count in long training log durations (leggedrobotics#210) --------- Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * Fix the CNNModel to support 2D observations only (leggedrobotics#215) * Bump version to 5.4.1 --------- Co-authored-by: Clemens Schwarke <96480707+ClemensSchwarke@users.noreply.github.com> Co-authored-by: adenzler-nvidia <adenzler@nvidia.com> Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> Co-authored-by: Emilio Palma <115191961+epalmaEth@users.noreply.github.com> Co-authored-by: epalmaEth <epalma@student.ethz.ch> Co-authored-by: Kevin Zakka <kevinzakka@users.noreply.github.com> Co-authored-by: Julia Beliaeva <juliabeliaeva@users.noreply.github.com> Co-authored-by: Masahiro Kubota <42679530+masahiro-kubota@users.noreply.github.com>
aj-persona
pushed a commit
to PAI-IHMC/rsl_rl
that referenced
this pull request
Jul 9, 2026
* Support direct class passing (leggedrobotics#145) Co-authored-by: Lorenzo Terenzi <lorenzoterenzi96@gmail.com> * Fixed a small typo in the cnn.py (leggedrobotics#153) Change weight initialization statement from Xavier to Kaiming * update license header * Separates actor and critic (leggedrobotics#159) * Move functions from Runners to Algorithms (leggedrobotics#164) Approved in ClemensSchwarke#4 * Fix multi-gpu initialization with W&B logging (leggedrobotics#166) * Add video logging for WandB (leggedrobotics#167) * Add model print outs (leggedrobotics#168) * Fixes training with partially recurrent architectures (leggedrobotics#169) * Add model exports to ONNX and JIT files (leggedrobotics#170) * Adds option to share CNN encoders between actor and critic models (leggedrobotics#171) * Bump version to 4.0.0 * Fix CNNModel init ordering bugs (leggedrobotics#175) * Bump version to 4.0.1 * log code state inside init_logging_writer (leggedrobotics#178) * log code state inside init_logging_writer * remove self and update comments --------- Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * fix multi gpu unbounded growth due to logging (leggedrobotics#179) --------- Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * remove resolved todos * Add commit hash to repo diff files * Adds a batch class to avoid positional arguments in generators (leggedrobotics#172) * Fix three bugs: missing raise, wrong RND index, variable typo (leggedrobotics#180) 1. rnn.py: Add missing `raise` before NotImplementedError -- the exception was being constructed but never raised, causing silent failure when resetting hidden state of done environments with a custom hidden state. 2. ppo.py: Fix wrong index in broadcast_parameters -- when RND is enabled, the predictor was loading model_params[1] (critic state) instead of model_params[2] (its own state), corrupting RND weights during multi-GPU training. 3. cnn_model.py: Fix variable name typo `latend_cnn` -> `latent_cnn`. * Fix mutable default arguments and replace assertions with proper exceptions (leggedrobotics#181) - Replace mutable list default `[256, 256, 256]` with tuple `(256, 256, 256)` in MLPModel, CNNModel, and RNNModel to prevent potential shared state across instances. - Replace `assert` statements used for input validation with `raise ValueError(...)` in MLPModel, CNNModel, and RND, since assertions are silently stripped when running with `python -O`. * Add a distribution class for easier adaptability (leggedrobotics#173) * Adds a small documentation (leggedrobotics#183) * add docstrings to all functions * add sphinx docs * Adds tests to the library (leggedrobotics#184) * add tests * add onnx tests * add on_policy_runner test * add test workflow * add lint workflow * fix onnx test failure * Adds NAN check to avoid ambiguous std >= 0.0 error (leggedrobotics#185) * Bump version to 5.0.0 * Fix shared CNN modules in _OnnxCNNModel and _TorchCNNModel (leggedrobotics#189) Deep-copy CNN modules in export wrappers to prevent shared state with the original training model. Fixes leggedrobotics#188. * Fix _OnnxCNNModel forward signature for ONNX export (leggedrobotics#187) * Bump version to 5.0.1 * minor docs fixes * Add Torch compile for PPO and Distillation (leggedrobotics#199) --------- Co-authored-by: adenzler-nvidia <adenzler@nvidia.com> * Clean up PPO by moving extension logic to the extension files (leggedrobotics#200) * Bump version to 5.1.0 * Add clip and constant std functionalities to Gaussian Distribution (leggedrobotics#201) --------- Co-authored-by: epalmaEth <epalma@student.ethz.ch> Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * Logger: union keys across the window instead of sampling from index 0 (leggedrobotics#205) --------- Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * Save and restore the critic's hidden state in compute_returns (leggedrobotics#206) * Adds a Beta distribution (leggedrobotics#202) * Bump version to 5.3.0 * add numpy to mock imports for building the docs * Enable custom logging services (leggedrobotics#211) * Preserve observation tensors dtype in the rollout storage (leggedrobotics#212) --------- Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * fix: include day count in long training log durations (leggedrobotics#210) --------- Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> * Fix the CNNModel to support 2D observations only (leggedrobotics#215) * Bump version to 5.4.1 * initialize logger in runner constructor * set wandb entity from agent_cfg rather than os environment * change model save path to be under model dir * delete deprecated wandb_utils.py no longer exist in ver 5.4.1 * if class in given to symmetry augmentation callable, return instantiated object, not class. --------- Co-authored-by: Clemens Schwarke <96480707+ClemensSchwarke@users.noreply.github.com> Co-authored-by: Lorenzo Terenzi <lorenzoterenzi96@gmail.com> Co-authored-by: Wu Shenglin <wslin281411@163.com> Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com> Co-authored-by: Kevin Zakka <kevinzakka@users.noreply.github.com> Co-authored-by: Anke Zhao <1203302838@qq.com> Co-authored-by: Vineet <52542471+VineetTambe@users.noreply.github.com> Co-authored-by: Jash Shah <49280550+jashshah999@users.noreply.github.com> Co-authored-by: adenzler-nvidia <adenzler@nvidia.com> Co-authored-by: Emilio Palma <115191961+epalmaEth@users.noreply.github.com> Co-authored-by: epalmaEth <epalma@student.ethz.ch> Co-authored-by: Julia Beliaeva <juliabeliaeva@users.noreply.github.com> Co-authored-by: Masahiro Kubota <42679530+masahiro-kubota@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #130