Problem
ExperimentResult.episodes() reads all episode.metadata.json files eagerly on first access. load_all_trajectories() and load_all_trajectory_metadata() return list[Trajectory], loading everything into memory.
At 100k episodes, this is a significant memory and latency bottleneck.
Proposed Fix
- Make
ExperimentResult.episodes() yield EpisodeResult wrappers without reading metadata — the wrapper already loads metadata lazily on first .metadata() call.
- Return iterators instead of lists from
load_all_trajectories() and load_all_trajectory_metadata().
- Consider
ThreadPoolExecutor for parallel metadata reads when the full list is needed.
Context
Identified during review of PR #262 (Storage V2).
Problem
ExperimentResult.episodes()reads allepisode.metadata.jsonfiles eagerly on first access.load_all_trajectories()andload_all_trajectory_metadata()returnlist[Trajectory], loading everything into memory.At 100k episodes, this is a significant memory and latency bottleneck.
Proposed Fix
ExperimentResult.episodes()yieldEpisodeResultwrappers without reading metadata — the wrapper already loads metadata lazily on first.metadata()call.load_all_trajectories()andload_all_trajectory_metadata().ThreadPoolExecutorfor parallel metadata reads when the full list is needed.Context
Identified during review of PR #262 (Storage V2).