MNT: add timeout on batch jobs

[tech3371]

Change Summary

closes #768

Overview

I believe this should enable timeout on batch job to 2 hrs. I can test that next week since I will be out rest of this week.

Updated Files

• sds_data_manager/constructs/processing_construct.py
  • add timeout to batch job definition

Testing

[pleasant-menlo]

This looks good to us. Can we hold off on merging this until we have finished GLOWS validation? We suspect we will have some longer running jobs until validation is done. We don't expect to need anything close to this long once we are processing real data.

We will approve once we are finished with validation.

• Sebastian and Ethan

[greglucas]

This looks good to us. Can we hold off on merging this until we have finished GLOWS validation? We suspect we will have some longer running jobs until validation is done. We don't expect to need anything close to this long once we are processing real data.

@pleasant-menlo do you know if you need more compute / memory resources on your batch jobs? We could look into increasing some of those limits as well to speed up the processing as I think we are really pretty minimal at the moment.

[pleasant-menlo]

@pleasant-menlo do you know if you need more compute / memory resources on your batch jobs? We could look into increasing some of those limits as well to speed up the processing as I think we are really pretty minimal at the moment.

Hi @greglucas, We're not confident these would take more than 2 hours to on real data, but we have seen SWAPI take a while to run, and we predict the maps, and glows that are triggered through scheduled jobs may take a while to run as well. So these would be the ones to look out for in the future.