[BugFix] Fix schemes and refactor collectors to make them readable

[vmoens]

Stack from ghstack (oldest at bottom):

• -> [BugFix] Fix schemes and refactor collectors to make them readable #3226
• [Refactor] Non-daemonic processes in PEnv #3225

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3226

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• ROCm failures during provisioning step due to network issues

❌ 13 New Failures, 3 Unrelated Failures

As of commit 2d9fec3 with merge base 13434eb ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 139.
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 98af99fa663a11d216623ca3a3ff3261894e36ccda621122e862d4502fb06b90 /exec failed with exit code 1
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t aa631d95b705e3fbc8b9a682623178600d46bc9b0af5445c08fd3804cc820e77 /exec failed with exit code 1
• Push Binary Nightly / build-wheel-unix (linux, ubuntu-22.04, 3.9, cp39-cp39, cpu, cpu) (gh)
• Push Binary Nightly / build-wheel-unix (macos, macos-latest, 3.9, cp39-cp39, cpu, cpu) (gh)
• Push Binary Nightly / build-wheel-windows (3.9, 3.9) (gh)
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t c7815f571337a11d5a168b56f23f16005f668d3d7486c52aaec0c3b19d611336 /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  test/test_env.py::TestNonTensorEnv::test_parallel[False-True]
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh)
  test/test_env.py::TestNonTensorEnv::test_parallel[False-False]
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh)
  test/test_distributed.py::TestRayCollector::test_distributed_collector_updatepolicy[True-MultiaSyncDataCollector]
• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh)
  test/test_collector.py::TestPolicyFactory::test_weight_update[scheme_shared]
• Unit-tests on Linux / tests-optdeps (3.11, 12.8) / linux-job (gh)
  test/test_collector.py::TestCollectorGeneric::test_update_weights[SharedMemWeightSyncScheme-True-True]
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh)
  test/test_distributed.py::TestRayCollector::test_distributed_collector_updatepolicy[True-MultiaSyncDataCollector]

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Push Binary Nightly / test-wheel-unix (linux, ubuntu-22.04, 3.9, cp39-cp39, cpu, cpu) (gh) (trunk failure)
  Process completed with exit code 1.
• Push Binary Nightly / test-wheel-unix (macos, macos-latest, 3.9, cp39-cp39, cpu, cpu) (gh) (trunk failure)
  Process completed with exit code 1.
• Push Binary Nightly / test-wheel-windows (3.9, 3.9) (gh) (trunk failure)
  Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 154. Improved: $\large\color{#35bf28}21$. Worsened: $\large\color{#d91a1a}7$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_tensor_to_bytestream_speed[pickle]	82.6324μs	81.8645μs	12.2153 KOps/s	12.3822 KOps/s	$\color{#d91a1a}-1.35\%$
test_tensor_to_bytestream_speed[torch.save]	0.1387ms	0.1384ms	7.2261 KOps/s	7.1374 KOps/s	$\color{#35bf28}+1.24\%$
test_tensor_to_bytestream_speed[untyped_storage]	0.1162s	0.1159s	8.6295 Ops/s	8.5352 Ops/s	$\color{#35bf28}+1.11\%$
test_tensor_to_bytestream_speed[numpy]	2.7784μs	2.7679μs	361.2789 KOps/s	358.9466 KOps/s	$\color{#35bf28}+0.65\%$
test_tensor_to_bytestream_speed[safetensors]	41.6092μs	41.3385μs	24.1905 KOps/s	24.2113 KOps/s	$\color{#d91a1a}-0.09\%$
test_simple	0.5466s	0.5457s	1.8326 Ops/s	1.7509 Ops/s	$\color{#35bf28}+4.66\%$
test_transformed	1.2207s	1.1289s	0.8858 Ops/s	0.8880 Ops/s	$\color{#d91a1a}-0.25\%$
test_serial	1.6537s	1.6495s	0.6062 Ops/s	0.6074 Ops/s	$\color{#d91a1a}-0.18\%$
test_parallel	1.1709s	1.0974s	0.9112 Ops/s	0.9345 Ops/s	$\color{#d91a1a}-2.49\%$
test_step_mdp_speed[True-True-True-True-True]	0.1890ms	44.5309μs	22.4563 KOps/s	23.2326 KOps/s	$\color{#d91a1a}-3.34\%$
test_step_mdp_speed[True-True-True-True-False]	53.0900μs	25.3941μs	39.3792 KOps/s	41.0032 KOps/s	$\color{#d91a1a}-3.96\%$
test_step_mdp_speed[True-True-True-False-True]	67.1110μs	24.6107μs	40.6328 KOps/s	40.5851 KOps/s	$\color{#35bf28}+0.12\%$
test_step_mdp_speed[True-True-True-False-False]	43.4210μs	13.7694μs	72.6250 KOps/s	73.1174 KOps/s	$\color{#d91a1a}-0.67\%$
test_step_mdp_speed[True-True-False-True-True]	89.0410μs	47.7436μs	20.9452 KOps/s	21.1776 KOps/s	$\color{#d91a1a}-1.10\%$
test_step_mdp_speed[True-True-False-True-False]	53.0010μs	27.9254μs	35.8097 KOps/s	36.3719 KOps/s	$\color{#d91a1a}-1.55\%$
test_step_mdp_speed[True-True-False-False-True]	59.1910μs	27.5724μs	36.2682 KOps/s	36.2546 KOps/s	$\color{#35bf28}+0.04\%$
test_step_mdp_speed[True-True-False-False-False]	67.6820μs	16.7929μs	59.5491 KOps/s	60.8796 KOps/s	$\color{#d91a1a}-2.19\%$
test_step_mdp_speed[True-False-True-True-True]	89.8210μs	50.6436μs	19.7458 KOps/s	20.1310 KOps/s	$\color{#d91a1a}-1.91\%$
test_step_mdp_speed[True-False-True-True-False]	0.1049ms	30.1651μs	33.1509 KOps/s	33.0120 KOps/s	$\color{#35bf28}+0.42\%$
test_step_mdp_speed[True-False-True-False-True]	64.5510μs	27.1948μs	36.7717 KOps/s	36.1143 KOps/s	$\color{#35bf28}+1.82\%$
test_step_mdp_speed[True-False-True-False-False]	56.2110μs	15.9233μs	62.8009 KOps/s	61.2891 KOps/s	$\color{#35bf28}+2.47\%$
test_step_mdp_speed[True-False-False-True-True]	84.7820μs	52.5789μs	19.0190 KOps/s	19.0061 KOps/s	$\color{#35bf28}+0.07\%$
test_step_mdp_speed[True-False-False-True-False]	63.4510μs	32.6033μs	30.6718 KOps/s	30.0609 KOps/s	$\color{#35bf28}+2.03\%$
test_step_mdp_speed[True-False-False-False-True]	61.4310μs	29.7416μs	33.6229 KOps/s	32.9158 KOps/s	$\color{#35bf28}+2.15\%$
test_step_mdp_speed[True-False-False-False-False]	43.1210μs	19.2903μs	51.8395 KOps/s	51.8363 KOps/s	$+0.01\%$
test_step_mdp_speed[False-True-True-True-True]	99.1210μs	49.5473μs	20.1827 KOps/s	19.7323 KOps/s	$\color{#35bf28}+2.28\%$
test_step_mdp_speed[False-True-True-True-False]	0.1023ms	30.5997μs	32.6801 KOps/s	32.7387 KOps/s	$\color{#d91a1a}-0.18\%$
test_step_mdp_speed[False-True-True-False-True]	2.4072ms	32.0161μs	31.2343 KOps/s	32.1458 KOps/s	$\color{#d91a1a}-2.84\%$
test_step_mdp_speed[False-True-True-False-False]	46.5510μs	18.2892μs	54.6769 KOps/s	55.7418 KOps/s	$\color{#d91a1a}-1.91\%$
test_step_mdp_speed[False-True-False-True-True]	97.6510μs	52.7563μs	18.9551 KOps/s	19.1480 KOps/s	$\color{#d91a1a}-1.01\%$
test_step_mdp_speed[False-True-False-True-False]	75.1910μs	33.3421μs	29.9921 KOps/s	30.5658 KOps/s	$\color{#d91a1a}-1.88\%$
test_step_mdp_speed[False-True-False-False-True]	90.4510μs	34.5616μs	28.9339 KOps/s	29.6179 KOps/s	$\color{#d91a1a}-2.31\%$
test_step_mdp_speed[False-True-False-False-False]	51.6110μs	21.1365μs	47.3114 KOps/s	48.0739 KOps/s	$\color{#d91a1a}-1.59\%$
test_step_mdp_speed[False-False-True-True-True]	96.6510μs	55.5396μs	18.0052 KOps/s	18.2758 KOps/s	$\color{#d91a1a}-1.48\%$
test_step_mdp_speed[False-False-True-True-False]	92.1310μs	35.8850μs	27.8668 KOps/s	28.0610 KOps/s	$\color{#d91a1a}-0.69\%$
test_step_mdp_speed[False-False-True-False-True]	71.4610μs	34.0707μs	29.3507 KOps/s	29.6786 KOps/s	$\color{#d91a1a}-1.10\%$
test_step_mdp_speed[False-False-True-False-False]	48.8000μs	21.0013μs	47.6161 KOps/s	47.9368 KOps/s	$\color{#d91a1a}-0.67\%$
test_step_mdp_speed[False-False-False-True-True]	94.9220μs	57.3407μs	17.4396 KOps/s	17.3445 KOps/s	$\color{#35bf28}+0.55\%$
test_step_mdp_speed[False-False-False-True-False]	91.9020μs	38.0893μs	26.2541 KOps/s	25.9310 KOps/s	$\color{#35bf28}+1.25\%$
test_step_mdp_speed[False-False-False-False-True]	95.7120μs	35.8210μs	27.9166 KOps/s	27.9936 KOps/s	$\color{#d91a1a}-0.28\%$
test_step_mdp_speed[False-False-False-False-False]	59.0910μs	23.4474μs	42.6486 KOps/s	43.0823 KOps/s	$\color{#d91a1a}-1.01\%$
test_values[generalized_advantage_estimate-True-True]	9.9744ms	9.7730ms	102.3232 Ops/s	102.7654 Ops/s	$\color{#d91a1a}-0.43\%$
test_values[vec_generalized_advantage_estimate-True-True]	20.9622ms	17.6607ms	56.6230 Ops/s	56.8518 Ops/s	$\color{#d91a1a}-0.40\%$
test_values[td0_return_estimate-False-False]	0.2582ms	0.1310ms	7.6361 KOps/s	7.8110 KOps/s	$\color{#d91a1a}-2.24\%$
test_values[td1_return_estimate-False-False]	27.1951ms	26.4767ms	37.7691 Ops/s	37.9314 Ops/s	$\color{#d91a1a}-0.43\%$
test_values[vec_td1_return_estimate-False-False]	20.6745ms	17.9418ms	55.7358 Ops/s	56.6229 Ops/s	$\color{#d91a1a}-1.57\%$
test_values[td_lambda_return_estimate-True-False]	42.0657ms	39.5352ms	25.2939 Ops/s	25.5902 Ops/s	$\color{#d91a1a}-1.16\%$
test_values[vec_td_lambda_return_estimate-True-False]	20.6559ms	17.8328ms	56.0764 Ops/s	57.1485 Ops/s	$\color{#d91a1a}-1.88\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	8.4585ms	8.3735ms	119.4245 Ops/s	118.7082 Ops/s	$\color{#35bf28}+0.60\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	1.7651ms	1.4435ms	692.7562 Ops/s	674.2093 Ops/s	$\color{#35bf28}+2.75\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.4822ms	0.4211ms	2.3749 KOps/s	2.4752 KOps/s	$\color{#d91a1a}-4.05\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	35.0089ms	34.1206ms	29.3078 Ops/s	33.9918 Ops/s	$\textbf{\color{#d91a1a}-13.78\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	2.0901ms	1.7679ms	565.6484 Ops/s	579.1676 Ops/s	$\color{#d91a1a}-2.33\%$
test_dqn_speed[False-None]	6.4893ms	1.4236ms	702.4414 Ops/s	717.5972 Ops/s	$\color{#d91a1a}-2.11\%$
test_dqn_speed[False-backward]	2.0135ms	1.9172ms	521.5995 Ops/s	514.0727 Ops/s	$\color{#35bf28}+1.46\%$
test_dqn_speed[True-None]	0.6959ms	0.5096ms	1.9623 KOps/s	1.9387 KOps/s	$\color{#35bf28}+1.22\%$
test_dqn_speed[True-backward]	1.0133ms	0.9669ms	1.0342 KOps/s	856.7872 Ops/s	$\textbf{\color{#35bf28}+20.71\%}$
test_dqn_speed[reduce-overhead-None]	0.6563ms	0.5052ms	1.9794 KOps/s	1.9221 KOps/s	$\color{#35bf28}+2.98\%$
test_dqn_speed[reduce-overhead-backward]	1.1288ms	0.9506ms	1.0519 KOps/s	883.3358 Ops/s	$\textbf{\color{#35bf28}+19.08\%}$
test_ddpg_speed[False-None]	3.2190ms	2.8598ms	349.6711 Ops/s	339.6162 Ops/s	$\color{#35bf28}+2.96\%$
test_ddpg_speed[False-backward]	4.2851ms	4.0770ms	245.2770 Ops/s	244.8440 Ops/s	$\color{#35bf28}+0.18\%$
test_ddpg_speed[True-None]	1.7318ms	1.3677ms	731.1597 Ops/s	706.2236 Ops/s	$\color{#35bf28}+3.53\%$
test_ddpg_speed[True-backward]	2.4248ms	2.3516ms	425.2431 Ops/s	390.2855 Ops/s	$\textbf{\color{#35bf28}+8.96\%}$
test_ddpg_speed[reduce-overhead-None]	1.4997ms	1.3619ms	734.2601 Ops/s	713.4923 Ops/s	$\color{#35bf28}+2.91\%$
test_ddpg_speed[reduce-overhead-backward]	2.3855ms	2.3290ms	429.3734 Ops/s	420.8012 Ops/s	$\color{#35bf28}+2.04\%$
test_sac_speed[False-None]	8.2200ms	7.8269ms	127.7639 Ops/s	127.1747 Ops/s	$\color{#35bf28}+0.46\%$
test_sac_speed[False-backward]	11.6548ms	11.1079ms	90.0264 Ops/s	89.2774 Ops/s	$\color{#35bf28}+0.84\%$
test_sac_speed[True-None]	2.4512ms	2.0977ms	476.7055 Ops/s	470.2869 Ops/s	$\color{#35bf28}+1.36\%$
test_sac_speed[True-backward]	4.1568ms	4.0284ms	248.2365 Ops/s	246.5227 Ops/s	$\color{#35bf28}+0.70\%$
test_sac_speed[reduce-overhead-None]	2.4644ms	2.1132ms	473.2191 Ops/s	460.8024 Ops/s	$\color{#35bf28}+2.69\%$
test_sac_speed[reduce-overhead-backward]	4.1522ms	4.0467ms	247.1146 Ops/s	200.4324 Ops/s	$\textbf{\color{#35bf28}+23.29\%}$
test_redq_speed[False-None]	10.7571ms	10.1888ms	98.1467 Ops/s	95.2199 Ops/s	$\color{#35bf28}+3.07\%$
test_redq_speed[False-backward]	18.3669ms	17.5528ms	56.9709 Ops/s	54.9862 Ops/s	$\color{#35bf28}+3.61\%$
test_redq_speed[True-None]	4.6092ms	4.3976ms	227.3978 Ops/s	224.6425 Ops/s	$\color{#35bf28}+1.23\%$
test_redq_speed[True-backward]	11.5031ms	9.7709ms	102.3448 Ops/s	101.9167 Ops/s	$\color{#35bf28}+0.42\%$
test_redq_speed[reduce-overhead-None]	5.3354ms	4.4490ms	224.7695 Ops/s	236.4557 Ops/s	$\color{#d91a1a}-4.94\%$
test_redq_speed[reduce-overhead-backward]	10.3244ms	9.9757ms	100.2436 Ops/s	99.1834 Ops/s	$\color{#35bf28}+1.07\%$
test_redq_deprec_speed[False-None]	11.3359ms	10.9341ms	91.4569 Ops/s	91.2872 Ops/s	$\color{#35bf28}+0.19\%$
test_redq_deprec_speed[False-backward]	16.8071ms	15.7033ms	63.6807 Ops/s	63.6062 Ops/s	$\color{#35bf28}+0.12\%$
test_redq_deprec_speed[True-None]	4.0685ms	3.6285ms	275.5954 Ops/s	276.7979 Ops/s	$\color{#d91a1a}-0.43\%$
test_redq_deprec_speed[True-backward]	7.9111ms	7.5199ms	132.9798 Ops/s	129.7652 Ops/s	$\color{#35bf28}+2.48\%$
test_redq_deprec_speed[reduce-overhead-None]	4.0810ms	3.6121ms	276.8495 Ops/s	260.8011 Ops/s	$\textbf{\color{#35bf28}+6.15\%}$
test_redq_deprec_speed[reduce-overhead-backward]	8.0346ms	7.5654ms	132.1807 Ops/s	122.8784 Ops/s	$\textbf{\color{#35bf28}+7.57\%}$
test_td3_speed[False-None]	8.1897ms	7.8809ms	126.8886 Ops/s	126.6213 Ops/s	$\color{#35bf28}+0.21\%$
test_td3_speed[False-backward]	11.4096ms	10.7835ms	92.7344 Ops/s	93.0537 Ops/s	$\color{#d91a1a}-0.34\%$
test_td3_speed[True-None]	1.8413ms	1.7888ms	559.0431 Ops/s	562.6379 Ops/s	$\color{#d91a1a}-0.64\%$
test_td3_speed[True-backward]	4.9213ms	3.9080ms	255.8827 Ops/s	227.8085 Ops/s	$\textbf{\color{#35bf28}+12.32\%}$
test_td3_speed[reduce-overhead-None]	1.8007ms	1.7573ms	569.0591 Ops/s	557.6752 Ops/s	$\color{#35bf28}+2.04\%$
test_td3_speed[reduce-overhead-backward]	3.7030ms	3.6132ms	276.7619 Ops/s	243.8896 Ops/s	$\textbf{\color{#35bf28}+13.48\%}$
test_cql_speed[False-None]	30.2172ms	26.1701ms	38.2116 Ops/s	27.6902 Ops/s	$\textbf{\color{#35bf28}+38.00\%}$
test_cql_speed[False-backward]	39.5653ms	35.7892ms	27.9414 Ops/s	28.0506 Ops/s	$\color{#d91a1a}-0.39\%$
test_cql_speed[True-None]	12.6522ms	12.2901ms	81.3662 Ops/s	79.0042 Ops/s	$\color{#35bf28}+2.99\%$
test_cql_speed[True-backward]	18.8237ms	18.3870ms	54.3863 Ops/s	55.4660 Ops/s	$\color{#d91a1a}-1.95\%$
test_cql_speed[reduce-overhead-None]	12.8233ms	12.4028ms	80.6271 Ops/s	79.3218 Ops/s	$\color{#35bf28}+1.65\%$
test_cql_speed[reduce-overhead-backward]	18.7977ms	18.4617ms	54.1662 Ops/s	54.0990 Ops/s	$\color{#35bf28}+0.12\%$
test_a2c_speed[False-None]	5.9326ms	5.4177ms	184.5816 Ops/s	181.9816 Ops/s	$\color{#35bf28}+1.43\%$
test_a2c_speed[False-backward]	12.8261ms	11.9228ms	83.8729 Ops/s	82.7572 Ops/s	$\color{#35bf28}+1.35\%$
test_a2c_speed[True-None]	3.9374ms	3.7143ms	269.2328 Ops/s	265.3776 Ops/s	$\color{#35bf28}+1.45\%$
test_a2c_speed[True-backward]	8.8348ms	8.6413ms	115.7231 Ops/s	114.8644 Ops/s	$\color{#35bf28}+0.75\%$
test_a2c_speed[reduce-overhead-None]	3.7734ms	3.6689ms	272.5620 Ops/s	269.3752 Ops/s	$\color{#35bf28}+1.18\%$
test_a2c_speed[reduce-overhead-backward]	8.9361ms	8.7775ms	113.9278 Ops/s	113.8371 Ops/s	$\color{#35bf28}+0.08\%$
test_ppo_speed[False-None]	6.0380ms	5.8552ms	170.7888 Ops/s	166.8719 Ops/s	$\color{#35bf28}+2.35\%$
test_ppo_speed[False-backward]	12.7393ms	12.5327ms	79.7913 Ops/s	78.7549 Ops/s	$\color{#35bf28}+1.32\%$
test_ppo_speed[True-None]	3.7612ms	3.6371ms	274.9451 Ops/s	268.7187 Ops/s	$\color{#35bf28}+2.32\%$
test_ppo_speed[True-backward]	8.5525ms	8.3639ms	119.5608 Ops/s	110.0781 Ops/s	$\textbf{\color{#35bf28}+8.61\%}$
test_ppo_speed[reduce-overhead-None]	3.7627ms	3.6178ms	276.4146 Ops/s	273.7373 Ops/s	$\color{#35bf28}+0.98\%$
test_ppo_speed[reduce-overhead-backward]	8.9447ms	8.6711ms	115.3253 Ops/s	113.7082 Ops/s	$\color{#35bf28}+1.42\%$
test_reinforce_speed[False-None]	4.7655ms	4.5803ms	218.3285 Ops/s	212.8752 Ops/s	$\color{#35bf28}+2.56\%$
test_reinforce_speed[False-backward]	7.6708ms	7.4709ms	133.8531 Ops/s	132.9877 Ops/s	$\color{#35bf28}+0.65\%$
test_reinforce_speed[True-None]	4.1357ms	2.9706ms	336.6372 Ops/s	344.7374 Ops/s	$\color{#d91a1a}-2.35\%$
test_reinforce_speed[True-backward]	7.9202ms	7.6919ms	130.0070 Ops/s	128.6701 Ops/s	$\color{#35bf28}+1.04\%$
test_reinforce_speed[reduce-overhead-None]	3.0581ms	2.8464ms	351.3194 Ops/s	349.6563 Ops/s	$\color{#35bf28}+0.48\%$
test_reinforce_speed[reduce-overhead-backward]	8.1084ms	7.8727ms	127.0209 Ops/s	113.6845 Ops/s	$\textbf{\color{#35bf28}+11.73\%}$
test_iql_speed[False-None]	25.8192ms	20.0789ms	49.8036 Ops/s	48.6337 Ops/s	$\color{#35bf28}+2.41\%$
test_iql_speed[False-backward]	36.3611ms	30.7379ms	32.5331 Ops/s	32.4914 Ops/s	$\color{#35bf28}+0.13\%$
test_iql_speed[True-None]	8.7673ms	8.4610ms	118.1890 Ops/s	115.9320 Ops/s	$\color{#35bf28}+1.95\%$
test_iql_speed[True-backward]	17.0642ms	16.7851ms	59.5765 Ops/s	60.2587 Ops/s	$\color{#d91a1a}-1.13\%$
test_iql_speed[reduce-overhead-None]	8.7500ms	8.4892ms	117.7962 Ops/s	116.6614 Ops/s	$\color{#35bf28}+0.97\%$
test_iql_speed[reduce-overhead-backward]	17.3956ms	17.0765ms	58.5599 Ops/s	58.5759 Ops/s	$\color{#d91a1a}-0.03\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.1892ms	5.9660ms	167.6170 Ops/s	165.1724 Ops/s	$\color{#35bf28}+1.48\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.5036ms	0.2735ms	3.6570 KOps/s	2.8142 KOps/s	$\textbf{\color{#35bf28}+29.95\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.4621ms	0.2546ms	3.9276 KOps/s	2.9549 KOps/s	$\textbf{\color{#35bf28}+32.92\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	6.0862ms	5.6824ms	175.9833 Ops/s	175.0744 Ops/s	$\color{#35bf28}+0.52\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.0238ms	0.2689ms	3.7193 KOps/s	2.8772 KOps/s	$\textbf{\color{#35bf28}+29.27\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.4615ms	0.2496ms	4.0072 KOps/s	3.0382 KOps/s	$\textbf{\color{#35bf28}+31.89\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	1.4146ms	1.2326ms	811.3101 Ops/s	735.1379 Ops/s	$\textbf{\color{#35bf28}+10.36\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	1.3516ms	1.1429ms	874.9813 Ops/s	784.4105 Ops/s	$\textbf{\color{#35bf28}+11.55\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	9.7578ms	5.9860ms	167.0575 Ops/s	169.5594 Ops/s	$\color{#d91a1a}-1.48\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.8009ms	0.4197ms	2.3826 KOps/s	1.9945 KOps/s	$\textbf{\color{#35bf28}+19.46\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.6227ms	0.4003ms	2.4982 KOps/s	2.1930 KOps/s	$\textbf{\color{#35bf28}+13.92\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	5.8415ms	5.7447ms	174.0732 Ops/s	174.1034 Ops/s	$\color{#d91a1a}-0.02\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.9526ms	0.3799ms	2.6325 KOps/s	3.0544 KOps/s	$\textbf{\color{#d91a1a}-13.81\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.5990ms	0.3648ms	2.7411 KOps/s	3.9118 KOps/s	$\textbf{\color{#d91a1a}-29.93\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.8277ms	5.6175ms	178.0138 Ops/s	176.3938 Ops/s	$\color{#35bf28}+0.92\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.4482s	0.9467ms	1.0563 KOps/s	3.6308 KOps/s	$\textbf{\color{#d91a1a}-70.91\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.5548ms	0.3244ms	3.0822 KOps/s	3.5775 KOps/s	$\textbf{\color{#d91a1a}-13.85\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	6.0594ms	5.8495ms	170.9548 Ops/s	169.1081 Ops/s	$\color{#35bf28}+1.09\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	2.1430ms	0.4871ms	2.0529 KOps/s	2.0721 KOps/s	$\color{#d91a1a}-0.93\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.6854ms	0.4720ms	2.1187 KOps/s	2.0799 KOps/s	$\color{#35bf28}+1.86\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	6.6918ms	5.0543ms	197.8513 Ops/s	195.7032 Ops/s	$\color{#35bf28}+1.10\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	10.3593ms	2.1391ms	467.4869 Ops/s	489.2716 Ops/s	$\color{#d91a1a}-4.45\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	1.9871ms	1.0288ms	971.9848 Ops/s	857.3887 Ops/s	$\textbf{\color{#35bf28}+13.37\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.4395s	13.8336ms	72.2875 Ops/s	192.4784 Ops/s	$\textbf{\color{#d91a1a}-62.44\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	2.9021ms	1.3996ms	714.5109 Ops/s	75.6421 Ops/s	$\textbf{\color{#35bf28}+844.59\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	11.5290ms	1.2315ms	812.0067 Ops/s	848.8842 Ops/s	$\color{#d91a1a}-4.34\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	7.2431ms	5.2664ms	189.8841 Ops/s	183.5403 Ops/s	$\color{#35bf28}+3.46\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	8.9693ms	2.1848ms	457.7127 Ops/s	586.7163 Ops/s	$\textbf{\color{#d91a1a}-21.99\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	7.6763ms	1.3393ms	746.6548 Ops/s	748.2537 Ops/s	$\color{#d91a1a}-0.21\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	35.9273ms	32.9051ms	30.3904 Ops/s	30.1131 Ops/s	$\color{#35bf28}+0.92\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	18.9681ms	17.4677ms	57.2486 Ops/s	57.3786 Ops/s	$\color{#d91a1a}-0.23\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True]	36.8882ms	34.4124ms	29.0593 Ops/s	28.8564 Ops/s	$\color{#35bf28}+0.70\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False]	19.5405ms	17.8848ms	55.9135 Ops/s	56.4662 Ops/s	$\color{#d91a1a}-0.98\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True]	41.2253ms	36.4212ms	27.4566 Ops/s	27.6353 Ops/s	$\color{#d91a1a}-0.65\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False]	20.7228ms	19.4119ms	51.5147 Ops/s	53.8354 Ops/s	$\color{#d91a1a}-4.31\%$