Feature to allow configurable dag retry and dag time out on airflow

[irainia]

Description

With this feature, the user could define the number of retry and the number of time out in their job spec. This allows more flexibility to the user, especially if there is a valid-long-running job happening.

Solution
Under job spec metadata, with airflow specific config, add two more fields (field name is discussable):

• Retry
• Timeout

We will need to update base_dag.py as well to accommodate this feature. Along with this, we have to ensure the order of assignment is fulfilled.

• if the fields are configured in the job spec, then we will use it regardless of the env
• if the fields are not configured, then we will use the env
• if the env is not set, then we will use default values