wait_for_completion in EmrCreateJobFlorOperator should wait for the cluster to complete to return success #61180
Description
Description
When wait_for_completion=True in EmrCreateJobFlowOperator, the operator code does not actually wait for the cluster to complete successfully before returning a success state. Instead, a success state is returned as soon as the cluster starts running. This can result in the task succeeding even if the cluster is terminated with errors after it begins running.
I believe this is due to this line of code that assigns the "WAIT_FOR_COMPLETION" WaitPolicy to the waiter. This corresponds to the "job_for_waiting" wait policy with which the waiter will only wait for the cluster to start running before returning a success state.
If the user wants the waiter to wait until the cluster completes, WAIT_FOR_STEPS_COMPLETION corresponding to the "job_flow_terminated" wait policy must be used. The operator has hard coded the "job_for_waiting" wait policy, so the user cannot configure the wait policy.
I propose adding a wait_policy parameter to the operator which allows the user to specify which wait policy they would prefer to use.
Use case/motivation
No response
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct