Update batch job retry strategy

[lacoak21]

Change Summary

closes #1472

Overview

We no longer need the "spot task was interrupted" retry because we are using on demand instances now. I also added a retry when the exit code is 75. This is the exit code that specifies this job should be retried. For example, when there was an imap-data-access 503 error we should retry the job. We should also retry the job when the job could not pull the container. E.g the status reason was "CannotPullContainerError".

File changes

• sds_data_manager/orchestration/imap_job.py
  • Update retry strategy

[Copilot]

🟡 Changes recommended

The new CannotPullContainerError* retry rule likely targets the wrong AWS Batch field (onReason vs onStatusReason), which could prevent the intended retries from occurring.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

Updates the AWS Batch retry strategy used by the IMAP orchestration job submission logic, aligning retries with current infrastructure (on-demand) and adding retry conditions for known transient failure modes related to reprocessing.

Changes:

• Replaces the Spot interruption-based retry condition with a retry on exit code 75 (explicit “retry me” exit code).
• Adds a retry rule intended to handle container image pull failures (CannotPullContainerError*).

File summaries

File	Description
sds_data_manager/orchestration/imap_job.py	Adjusts AWS Batch evaluateOnExit retry rules to target exit code 75 and container pull errors.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 2
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[bryan-harter]

Seems good to me, as long as the AI comment turns out to be incorrect/not relevant

[lacoak21]

Seems good to me, as long as the AI comment turns out to be incorrect/not relevant

I think it is relevant but I did change it to onStatusReason