[Feature] Reliability of the `watcher` execution mode.

[maoxingda]

Description

In watcher execution mode, the build of all models is placed into a single producer task. If a temporary, retryable, and recoverable error occurs during the build of a certain model, the execution mode falls back to local execution mode, and local execution mode has lower performance. How should this scenario be handled? Is there any plan in the community to address this?

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

• Yes, I am willing to submit a PR!

[tatiana]

Hi @maoxingda , thanks for reaching out!

We have a proposal to improve retries in #1978 (comment). What do you think of the idea? Do you have any other suggestions?

If the community doesn't contribute, Astronomer will be implementing the mentioned strategy within the next 1.5 months.

[maoxingda]

Hi, the screenshot idea is really, really great — looking forward to seeing it implemented.