[BugFix] Replay Buffer prefetch & SliceSampler #4738

Job	Run time
generate-matrix / generate	5s
pytorch/rl (pytorch/rl, test/smoke_test.py, torchrl, .github/scripts/pre-build-script.sh) / build-wheel-py3_10-cpu-aarch64	6m 15s
pytorch/rl (pytorch/rl, test/smoke_test.py, torchrl, .github/scripts/pre-build-script.sh) / upload / upload-wheel-py3_10-cpu-aarch64	31s
	6m 51s

Provide feedback