bug with PrioritizedEpisodeReplayBuffer

[FastWind123]

What happened + What you expected to happen

I got the following error.
Failure # 1 (occurred at 2026-02-04_20-36-47)
�[36mray::DQN.train()�[39m (pid=2226564, ip=172.17.0.4, actor_id=25e1dce4913c5ba94450e6ee01000000, repr=DQN(env=ale_py:ALE/Alien-v5; env-runners=14; learners=1; multi-agent=False))
File "/root/anaconda3/envs/rllib/lib/python3.13/site-packages/ray/tune/trainable/trainable.py", line 331, in train
raise skipped from exception_cause(skipped)
File "/root/anaconda3/envs/rllib/lib/python3.13/site-packages/ray/tune/trainable/trainable.py", line 328, in train
result = self.step()
File "/root/anaconda3/envs/rllib/lib/python3.13/site-packages/ray/rllib/algorithms/algorithm.py", line 1242, in step
train_results, train_iter_ctx = self._run_one_training_iteration()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/root/anaconda3/envs/rllib/lib/python3.13/site-packages/ray/rllib/algorithms/algorithm.py", line 3666, in _run_one_training_iteration
training_step_return_value = self.training_step()
File "/root/anaconda3/envs/rllib/lib/python3.13/site-packages/ray/rllib/algorithms/dqn/dqn.py", line 646, in training_step
return self._training_step_new_api_stack()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/root/anaconda3/envs/rllib/lib/python3.13/site-packages/ray/rllib/algorithms/dqn/dqn.py", line 668, in _training_step_new_api_stack
self.local_replay_buffer.add(episodes)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/root/anaconda3/envs/rllib/lib/python3.13/site-packages/ray/rllib/utils/replay_buffers/prioritized_episode_buffer.py", line 257, in add
episodes[new_episode_ids.index(eps_evicted_idxs[-1])]
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
ValueError: 0 is not in list

Here is my running code:

    .env_runners(
        # Every 4 agent steps a training update is performed.
        rollout_fragment_length=4,
        num_env_runners=ray_args.num_env_runners, # 1
        num_envs_per_env_runner=ray_args.num_envs_per_env_runner,  # yyt. vectorized envs per runner
        # env_to_module_connector=_make_env_to_module_connector, # pure bug, fuck.
        num_gpus_per_env_runner=ray_args.num_gpus_per_env_runner,  # yyt. disable env runner gpu usage. 0
    )

When I set num_envs_per_env_runner to a very high value, like 16, 32, it will raise this error.

Versions / Dependencies

ray==2.53.0

Reproduction script

See above

Issue Severity

None