Skip to content

handle chunking of long lists of job submissions for users #197

@jhkennedy

Description

@jhkennedy

Currently, HyP3 only allows submitting up to 200 jobs at a time.

Most of the SDK submit functions only submit a single job, but vail the prepare methods a long list of prepared jobs can be created and submitted via submit_prepared_jobs.

The SDK provides a helper function to "chunk" submissions into sets of 200 jobs for long submissions:
https://github.com/ASFHyP3/hyp3-sdk/blob/develop/src/hyp3_sdk/util.py#L40
and error handling needs to happen for each submitted chunck.

Therefore, typical user code for submitting a large number of jobs may look like:

import hyp3_sdk as sdk
import hyp3_sdk.util
import hyp3_sdk.exceptions

hyp3 = sdk.HyP3()

jobs = [...]  # A list of jobs > 200 in length

submitted_jobs = sdk.Batch()
failed_to_submit_jobs = []
for batch in sdk.util.chunk(jobs):
    try:
        submitted_jobs += hyp3.submit_prepared_jobs(batch)
    except sdk.exeptions.HyP3SDKError:
        failed_to_submit_jobs.extend(batch)
        print('Some failure message')

It would be nice to handle the chunking for users directly in the submit_prepared_jobs function. We have yet to do so up to this point because error handling is problematic in this space (e.g., do we return a list of failed jobs too?) and we've not determined a good way to handle failures.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions