Resumable checkpointable dataloading#1570

Open

pzelasko wants to merge 31 commits into

feat/resumable-checkpointable-dataloading

Commits on Mar 3, 2026

feat: Integrate sampler & IterableDatasetWrapper with checkpoint support

pzelasko
and
claude
committed
feat: Add CLI commands for creating binary index files

pzelasko
and
claude
committed
docs: Add resumable stateful dataloading documentation and torchdata dep

pzelasko
and
claude
committed
feat: Add StatefulDataLoader e2e tests and fix load_state_dict for torchdata

pzelasko
and
claude
committed
feat: Add Shar O(1) random access, IdentityDataset, shuffle API, and pickling fixes

pzelasko
and
claude
committed
fix: Address code review issues and add multi-node E2E resume test

pzelasko
and
claude
committed

Commits on Mar 4, 2026

feat: Split LazyIndexedSharIterator, add from_files indexed+shuffle, extend tutorial

pzelasko
and
claude
committed

Commits on Mar 5, 2026

feat: Support custom index_path for .idx files stored separately from data

pzelasko
and
claude
committed

Commits on Mar 7, 2026

Fix indexed reader restore and remote index caching
pzelasko
committed

Commits on May 1, 2026

Commits on May 5, 2026

Commits on May 6, 2026

fill documentation gaps
pzelasko
committed

Commits on May 12, 2026

Commits on May 13, 2026

Commits on May 28, 2026

Add ability to read byte ranges directly from AIStore for individual GET operations
pzelasko
committed

Commits on Jun 12, 2026