Skip to content

Should we provide an ordered=True option? #5

Open
@njsmith

Description

@njsmith

Right now, the incremental result reporting always reports results as they're ready, so the order of the output doesn't necessarily match the order of the input. (Except for run_all, since it's not incremental anyway.)

Should we provide an option ordered=True to request that the output order must match the input order?

If we do then there's some subtle interaction with backpressure: if task 0 is slow, we probably don't want to buffer up results from tasks 1...10000 while waiting for task 0 to finish. I think the way to avoid this is to do the reordering directly inside the workers, rather than with a separate buffering/reordering task. When ordered=True, have each worker wait for the previous work to report its result before this worker reports its result. And worker tasks still count against the metered limits until after they report their result.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions