Support inserting based on batchsize into shuffler#1369

ayushdg wants to merge 3 commits intoNVIDIA-NeMo:mainNVIDIA-NeMo/Curator:mainfrom

ayushdg:batched-exact-dedupayushdg/NeMo-Curator:batched-exact-dedupCopy head branch name to clipboard