Retry processing on AWS error

[pleasant-menlo]

Summary

Sometimes launching a batch job fails with an error in AWS, i.e.

"CannotPullContainerError: failed to resolve ref 593025701104.dkr.ecr.us-west-2.amazonaws.com/codice-l3-repo:latest for schema1 conversion: failed to do request: Head "https://593025701104.dkr.ecr.us-west-2.amazonaws.com/v2/codice-l3-repo/manifests/latest": dial tcp 16.145.66.67:443: i/o timeout"

It appears that the job is lost, and is not automatically retried. It could be nice if such situations were handled by retrying the job.

Error source

Requirements

• Detect when a job fails for an internal AWS reason
• Requeue the job automatically
• Limit to a certain number of retries

Related tickets

No response

[pleasant-menlo]

Dagster currently refuses to reprocess such jobs, saying "This exact job has been submitted previously, and has already failed. No need to run it again."

We may be able to work around this by implementing something similar to:

sds-data-manager/sds_data_manager/orchestration/imap_job.py

Lines 52 to 61 in 288ffce

	BATCH_JOB_RETRY_STRATEGY = {
	"attempts": 10,
	"evaluateOnExit": [
	{
	"onStatusReason": "Your Spot Task was interrupted.",
	"action": "RETRY",
	},
	{"onReason": "*", "action": "EXIT"},
	],
	}

Otherwise, is there a way to tell Dagster that some failures are not deterministic, and it is allowed to rerun the job?

[lacoak21]

This was completed in #1550