Trpo tianshou#585

Open

ShangYizhan wants to merge 587 commits intomicrosoft:v0.3from

ShangYizhan:trpo_tianshou

This pull request is big! We're only showing the most recent 250 commits

Commits on Jun 1, 2021

Commits on Jun 3, 2021

Commits on Jun 4, 2021

1. added parallel policy manager prototype; 2. used training ep for evaluation episodes
ysqyang
committed

Commits on Jun 9, 2021

refined parallel policy manager
ysqyang
committed
updated rl/__init__/py
ysqyang
committed
fixed lint issues and CIM local learner bugs
ysqyang
committed
deleted unwanted supply_chain test files
ysqyang
committed
revised default config for cim-dqn
ysqyang
committed

Commits on Jun 10, 2021

removed test_store.py as it is no longer needed
ysqyang
committed

Commits on Jun 11, 2021

1. changed Actor class to rollout_worker function; 2. renamed algorithm to algorithms
ysqyang
committed
updated figures
ysqyang
committed
removed unwanted import
ysqyang
committed

Commits on Jun 15, 2021

refactored CIM-DQN example
ysqyang
committed

Commits on Jun 16, 2021

added MultiProcessRolloutManager and MultiProcessTrainingManager
ysqyang
committed

Commits on Jun 17, 2021

updated doc
ysqyang
committed

Commits on Jun 18, 2021

lint issue fix
ysqyang
committed
lint issue fix
ysqyang
committed
fixed import formatting
ysqyang
committed
[Feature] Prioritized Experience Replay (#355)

ysqyang
and
ysqyang
authored
rm AbsDecisionGenerator
ysqyang
committed
Merge branch 'v0.2_rl_refinement' of github.com:microsoft/maro into v0.2_rl_refinement
ysqyang
committed
small fixes
ysqyang
committed
bug fix
ysqyang
committed

Commits on Jun 20, 2021

reorganized training folder structure
ysqyang
committed
fixed lint issues
ysqyang
committed
fixed lint issues
ysqyang
committed

Commits on Jun 21, 2021

policy manager refined
ysqyang
committed
lint fix
ysqyang
committed
restructured CIM-dqn sync code
ysqyang
committed

Commits on Jun 22, 2021

added policy version index and used it as a measure of experience staleness
ysqyang
committed
lint issue fix
ysqyang
committed
lint issue fix
ysqyang
committed
switched log_dir and proxy_kwargs order
ysqyang
committed

Commits on Jun 23, 2021

cim example refinement
ysqyang
committed

Commits on Jun 28, 2021

eval schedule sorted only when it's a list
ysqyang
committed
eval schedule sorted only when it's a list
ysqyang
committed
update sc env wrapper
Jinyu-W
committed
added docker scripts for cim-dqn
ysqyang
committed

Commits on Jun 29, 2021

Merge branch 'master' into v0.2
Jinyu-W
committed
refactored example folder structure and added workflow templates
ysqyang
committed
fixed merge conflicts
ysqyang
committed

Commits on Jun 30, 2021

fixed lint issues
ysqyang
committed
fixed lint issues
ysqyang
committed
fixed template bugs
ysqyang
committed
removed unused imports
ysqyang
committed
refactoring sc in progress
ysqyang
committed
simplified cim meta
ysqyang
committed
updated sc code
ysqyang
committed
fixed build.sh path bug
ysqyang
committed
refined sc and template code
ysqyang
committed
template refinement
ysqyang
committed
fixed merge conflicts
ysqyang
committed

Commits on Jul 1, 2021

deleted obsolete svgs
ysqyang
committed
merged with remote
ysqyang
committed
updated learner logs
ysqyang
committed
minor edits
ysqyang
committed
refactored templates for easy merge with async PR
ysqyang
committed
added component names for rollout manager and policy manager
ysqyang
committed
fixed incorrect position to add last episode to eval schedule
ysqyang
committed
added max_lag option in templates
ysqyang
committed
formatting edit in docker_compose_yml script
ysqyang
committed
moved local learner and early stopper outside sync_tools
ysqyang
committed
refactored rl toolkit folder structure
ysqyang
committed

Commits on Jul 2, 2021

refactored rl toolkit folder structure
ysqyang
committed
moved env_wrapper and agent_wrapper inside rl/learner
ysqyang
committed
refined scripts
ysqyang
committed
modified sc imports according to changes in rl toolkit folder structure
ysqyang
committed
fixed typo in script
ysqyang
committed
changes needed for running sc
ysqyang
committed
removed unwanted imports
ysqyang
committed
Merge branch 'v0.2_rl_refinement' into v0.2_rl_refinement_sc
ysqyang
committed
config change for testing sc scenario
ysqyang
committed

Commits on Jul 4, 2021

changes for perf testing
ysqyang
committed

Commits on Jul 5, 2021

Commits on Jul 6, 2021

added sleep in distributed launcher to avoid hanging
ysqyang
committed

Commits on Jul 7, 2021

updated api doc and rl toolkit doc
ysqyang
committed
refined dynamic imports using importlib
yaqiu
committed

Commits on Jul 8, 2021

Merge branch 'master' into v0.2

Jinyu Wang
authored and
Jinyu Wang
committed
1. moved policy update triggers to policy manager; 2. added version control in policy manager
yaqiu
committed
fixed a few bugs and updated cim RL example
yaqiu
committed
fixed a few more bugs
yaqiu
committed
resolved merge conflicts
yaqiu
committed

Commits on Jul 9, 2021

Commits on Jul 11, 2021

Commits on Jul 12, 2021

Commits on Jul 14, 2021

added post-step, post-collect, post-eval and post-update callbacks
yaqiu
committed
fixed lint issues
yaqiu
committed
fixed lint issues
yaqiu
committed

Commits on Jul 15, 2021

fixed some bugs
yaqiu
committed
moved instantiation of policy manager inside simple learner
yaqiu
committed
Merge branch 'v0.2_rl_refinement' into v0.2_rl_refinement_sc
yaqiu
committed
fixed env_wrapper get_reward signature
yaqiu
committed
minor edits
yaqiu
committed
Merge branch 'v0.2_rl_refinement' into v0.2_rl_refinement_sc
yaqiu
committed
removed get_eperience kwargs from env_wrapper
yaqiu
committed
Merge branch 'v0.2_rl_refinement' into v0.2_rl_refinement_sc
yaqiu
committed
1. renamed step_callback to post_step in env_wrapper; 2. added get_eval_env_func to RL workflows
yaqiu
committed
Merge branch 'v0.2_rl_refinement' into v0.2_rl_refinement_sc
yaqiu
committed
added rollout exp disribution option in RL examples
yaqiu
committed
fixed merge conflicts
yaqiu
committed

Commits on Jul 19, 2021

1. added missing model.eval() to choose_action; 2.added entropy features to AC
yaqiu
committed
fixed a bug in ac entropy
yaqiu
committed
abbreviated coefficient to coeff
yaqiu
committed

Commits on Jul 22, 2021

removed -dqn from job name in rl example config
yaqiu
committed
added tmp patch to dev.df
yaqiu
committed
renamed image name for running rl examples
yaqiu
committed

Commits on Jul 28, 2021

added get_loss interface for core policies
yaqiu
committed

Commits on Jul 30, 2021

Commits on Aug 3, 2021

refactored policy and algorithms
yaqiu
committed

Commits on Aug 5, 2021

policy interface redesigned
ysqyang
committed

Commits on Aug 8, 2021

refined policy interfaces
ysqyang
committed
fixed typo
yaqiu
committed

Commits on Aug 9, 2021

fixed bugs in refactored policy interface
yaqiu
committed
fixed some bugs
yaqiu
committed

Commits on Aug 11, 2021

refactoring in progress
yaqiu
committed

Commits on Aug 17, 2021

policy interface and policy manager redesigned
yaqiu
committed

Commits on Aug 18, 2021

1. fixed bugs in ac and pg; 2. fixed bugs rl workflow scripts
yaqiu
committed
fixed bug in distributed policy manager
yaqiu
committed
fixed lint issues
yaqiu
committed
fixed lint issues
yaqiu
committed
added scipy in setup
yaqiu
committed

Commits on Aug 19, 2021

1. trimmed rollout manager code; 2. added option to docker scripts
yaqiu
committed

Commits on Aug 20, 2021

updated api doc for policy manager
yaqiu
committed

Commits on Aug 23, 2021

Commits on Aug 25, 2021

further rl toolkit code simplifications
yaqiu
committed

Commits on Aug 26, 2021

more numpy-based optimization in RL toolkit
yaqiu
committed

Commits on Aug 27, 2021

moved replay buffer inside policy
yaqiu
committed
bug fixes
yaqiu
committed

Commits on Aug 29, 2021

numpy optimization and associated refactoring
yaqiu
committed

Commits on Aug 31, 2021

extracted shaping logic out of env_sampler
ysqyang
committed
fixed bug in CIM shaping and lint issues
yaqiu
committed

Commits on Sep 1, 2021

preliminary implemetation of parallel batch inference
yaqiu
committed

Commits on Sep 2, 2021

fixed bug in ddpg transition recording
ysqyang
committed
put get_state, get_env_actions, get_reward back in EnvSampler
yaqiu
committed

Commits on Sep 5, 2021

simplified exploration and core model interfaces
yaqiu
committed

Commits on Sep 6, 2021

bug fixes and doc update
yaqiu
committed
added improve() interface for RLPolicy for single-thread support
yaqiu
committed

Commits on Sep 7, 2021

fixed simple policy manager bug
yaqiu
committed

Commits on Sep 11, 2021

updated doc, rst, notebook
yaqiu
committed
updated notebook
yaqiu
committed
fixed lint issues
yaqiu
committed

Commits on Sep 12, 2021

fixed entropy bugs in ac.py
yaqiu
committed
reverted to simple policy manager as default
yaqiu
committed

Commits on Sep 14, 2021

Commits on Sep 15, 2021

removed obsolete images
ysqyang
committed
Merge branch 'v0.2_rl_refinement' of github.com:microsoft/maro into v0.2_rl_refinement
ysqyang
committed
added back agent2policy for general workflow use
yaqiu
committed

Commits on Sep 16, 2021

V0.2 rl refinement dist (#377)
buptchan
authored

Commits on Sep 17, 2021

Commits on Sep 22, 2021

Commits on Sep 24, 2021

Commits on Sep 26, 2021

VM scheduling with RL (#375)

authored
SC refinement (#397)

lihuoran
and
Jinyu Wang
authored

Commits on Oct 9, 2021

refined workflow scripts
yaqiu
committed
fixed bug in ParallelAgentWrapper
yaqiu
committed

Commits on Oct 10, 2021

1. fixed lint issues; 2. refined main script in workflows
yaqiu
committed
lint issue fix
yaqiu
committed
restored default config for rl example
yaqiu
committed
Update rollout.py
ysqyang
authored

Commits on Oct 11, 2021

refined env var processing in policy manager workflow
yaqiu
committed

Commits on Oct 12, 2021

added hasattr check in agent wrapper
yaqiu
committed
updated docker_compose_yml.py
yaqiu
committed

Commits on Oct 13, 2021

Minor refinement
lihuoran
committed

Commits on Oct 14, 2021

Merge branch 'v0.2_rl_refinement' into v0.3
lihuoran
committed

Commits on Dec 6, 2021

Minor PR. Prepare to merge latest master branch into v0.3 branch. (#412)
lihuoran
authored

Commits on Dec 8, 2021

Merge latest master into v0.3 (#426)

authored

Commits on Jan 24, 2022

Change `Env.set_seed()` logic (#456)
lihuoran
authored

Commits on Mar 4, 2022

Remove all SC related files (#473)
lihuoran
authored

Commits on Mar 7, 2022

RL Toolkit V3 (#471)

authored

Commits on Mar 9, 2022

RL renaming v2 (#476)
lihuoran
authored

Commits on Mar 31, 2022

Cherry pick latest RL (#498)
lihuoran
authored

Commits on Apr 22, 2022

Cherry pick RL changes from `sc_refinement` (latest commit: `2a4869`) (#509)
lihuoran
authored

Commits on Apr 24, 2022

RL incremental refactor (#501)
lihuoran
authored

Commits on May 10, 2022

RL component bundle (#513)
lihuoran
authored

Commits on May 16, 2022

Add method to get mapping of available tick to frame index (#415)
chaosddp
authored

Commits on May 18, 2022

Cherry pick from sc_refinement (#527)
lihuoran
authored

Commits on May 25, 2022

Refine `terminal` / `next_agent_state` logic (#531)
lihuoran
authored

Commits on May 31, 2022

Merge master into v0.3 (#536)

authored

Commits on Jun 9, 2022

Merge master into v0.3 (#545)

authored

Commits on Jun 10, 2022

Merge branch 'master' into v0.3
Jinyu-W
committed

Commits on Jun 14, 2022

Merge branch 'master' into v0.3
Jinyu-W
committed

Commits on Jun 23, 2022

Update requirements. (#552)
lihuoran
authored

Commits on Aug 26, 2022

Commits on Aug 31, 2022

Refine decision event logic (#559)
lihuoran
authored

Commits on Dec 27, 2022

Refine rl component bundle (#549)
lihuoran
authored
merge master into v0.3
Jinyu Wang
committed
:Merge branch 'master' into v0.3
Jinyu Wang
committed

Commits on Jan 11, 2023

Remove numpy data type (#571)
lihuoran
authored

Commits on Feb 5, 2023

trpo
YizhanShang
committed

Commits on Feb 20, 2023

trpo
ShangYizhan
committed

Commits on Feb 22, 2023

trpo
ShangYizhan
committed

Commits on Feb 27, 2023