Skip to content

Duplicate data returned when use num_workers param (multi-processing) in Dataloader  #150

Description

@Albert-Ma

`trainloader = mz.dataloader.DataLoader(
dataset=trainset,
stage='train',
num_workers=30,
callback=padding_callback
)``

Result:

image

It shows that model will train num_worker time, 30 in this experiment, in one epoch!

A Possible Solution: splitting workload across all workers in Dataset/ __iter__ function!

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions