EMR Serverless client should auto-refresh Spark UI URL as it expires for long running jobs

[juhai]

What's the issue?

With Dagster pipes EMR Serverless client, the spark UI URL is provided when the job changes to running state. The URL is only valid for a fixed period of time, around 1 hour. There should to be a timed refresh operation that gives a new URL every X minutes, possibly 15-30 minutes to ensure the current URL is valid until the next one is fetched.

This is the message when an expired link is used: The authorization token included in the request has expired. Please generate a new URL.

What did you expect to happen?

The URL provided in the dagster UI is valid and shows spark UI when clicked.

How to reproduce?

Have a spark job running in EMR Serverless for more than an hour and observe the link expiring and no access to Spark UI is possible after this.

Dagster version

0.12.2

Deployment type

Dagster Cloud

Deployment details

No response

Additional information

The dagster-aws version I am using is 0.28.2 but the current main branch has no handling for this situation in the _wait_for_completion method.

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[juhai]

There is an additional complication that the URL can be used only once. If you open the Spark UI and close the tab, you can't reopen it anymore. Also, every time you ask for a new URL, it invalidates the previous ones making this quite complicated to handle without excessive logging from the job. I may give it a go as I had a solution before dagster-pipes that was ok for me but involves quite a bit of logging.

[noy-solvin]

Hi @hellendag, @cmpadden
I've submitted PR #33941 that fixes this issue.
Would love a review whenever you get a chance. Happy to adjust anything!