Skip to content

Commit 795a99f

Browse files
Rick Warrenmeta-codesync[bot]
authored andcommitted
set_distributed_sampler_epoch for multidataset samplers (#1040)
Summary: Pull Request resolved: #1040 - TorchTNT automatically sets the `sampler` epoch to ensure sampling is not the same across batches. - However, the epoch was *not* being set for `batch_sampler`s, which are used in the `MultiDataset` case. See [a related (since resolved) issue for Pytorch Lightning](Lightning-AI/pytorch-lightning#13316). - Here I'm running `set_epoch` for the `batch_sampler` when applicable. Reviewed By: galrotem Differential Revision: D87585393 fbshipit-source-id: e629253b45418710cf8c0bba118af24e37e2633e
1 parent 0b1d2c3 commit 795a99f

1 file changed

Lines changed: 5 additions & 5 deletions

File tree

torchtnt/framework/_loop_utils.py

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -77,11 +77,11 @@ def _maybe_set_distributed_sampler_epoch(
7777
See: https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler
7878
"""
7979
# Set current training epoch for any DistributedSampler in dataloader
80-
if isinstance(dataloader, torch.utils.data.DataLoader) and isinstance(
81-
dataloader.sampler,
82-
_DistributedSampler,
83-
):
84-
dataloader.sampler.set_epoch(current_epoch)
80+
if isinstance(dataloader, torch.utils.data.DataLoader):
81+
if isinstance(dataloader.sampler, _DistributedSampler):
82+
dataloader.sampler.set_epoch(current_epoch)
83+
elif isinstance(dataloader.batch_sampler, _DistributedSampler):
84+
dataloader.batch_sampler.set_epoch(current_epoch)
8585

8686

8787
def _set_module_training_mode(

0 commit comments

Comments
 (0)