-
Notifications
You must be signed in to change notification settings - Fork 8
Description
Currently, HyP3 only allows submitting up to 200 jobs at a time.
Most of the SDK submit functions only submit a single job, but vail the prepare methods a long list of prepared jobs can be created and submitted via submit_prepared_jobs.
The SDK provides a helper function to "chunk" submissions into sets of 200 jobs for long submissions:
https://github.com/ASFHyP3/hyp3-sdk/blob/develop/src/hyp3_sdk/util.py#L40
and error handling needs to happen for each submitted chunck.
Therefore, typical user code for submitting a large number of jobs may look like:
import hyp3_sdk as sdk
import hyp3_sdk.util
import hyp3_sdk.exceptions
hyp3 = sdk.HyP3()
jobs = [...] # A list of jobs > 200 in length
submitted_jobs = sdk.Batch()
failed_to_submit_jobs = []
for batch in sdk.util.chunk(jobs):
try:
submitted_jobs += hyp3.submit_prepared_jobs(batch)
except sdk.exeptions.HyP3SDKError:
failed_to_submit_jobs.extend(batch)
print('Some failure message')It would be nice to handle the chunking for users directly in the submit_prepared_jobs function. We have yet to do so up to this point because error handling is problematic in this space (e.g., do we return a list of failed jobs too?) and we've not determined a good way to handle failures.