Add invalid_vrs_ids filtering

Summary:
X-link: facebookresearch/detectron2#5108


As title, with this diff, we verified the dataloading can be reproduced with `num_worker=2` and with RC6 data augmentations

Differential Revision: D49954475

Author: jiaxuzhu92

d2go/data/build.py

@@ -159,14 +159,15 @@ def build_weighted_detection_train_loader(
             )
         )
 
-    sampler = RepeatFactorTrainingSampler(repeat_factors)
+    sampler = RepeatFactorTrainingSampler(repeat_factors, seed=cfg.SEED)
 
     return build_batch_data_loader(
         dataset,
         sampler,
         cfg.SOLVER.IMS_PER_BATCH,
         aspect_ratio_grouping=cfg.DATALOADER.ASPECT_RATIO_GROUPING,
         num_workers=cfg.DATALOADER.NUM_WORKERS,
+        seed=cfg.SEED,
     )