Unable to load scheduler dashboard in SLURMRunner, but can in cluster

[gilmorethomas]

Describe the issue:
Thanks for your time in advance. I have created a simple "hello world" example of a SLURMRunner and SLURMCluster in my environment. I like the interface for the SLURMRunner instead of effectively needing to create wrappers around jobs in the SLURMCluster construct.

I dispatch my SLURMCluster job via sbatch (since my login node cannot run my scheduler) to a worker node (node-01), and then this dispatches additional jobs on my worker nodes (node[01-06]). When I do this, I am able to visit the scheduler dashboard, although I am seeing slightly weird behavior in job allocation (not the point of this post, I need to look into this more).

When I create my SLURMRunner (same as this example https://jobqueue.dask.org/en/stable/runners-overview.html), my jobs are getting allocated and run, but I am unable to load the scheduler dashboard. I get a 404 Page Not Found when I visit the scheduler link output by the client.dashboard_link and also in the scheduler.json file. This is not the same as when the runner spins down, as in this case I get the Connection Refused. Is this expected?

Minimal Complete Verifiable Example:
Using the SLURMRunner in my multi-node environment

# Put your MCVE code here

Anything else we need to know?:

Environment:

• Dask version: 2023.6.0
• Python version: 3.11.4
• Operating System: CENTOS-7
• Install method: pip

[jacobtomlinson]

This may just be a case where the client.dashboard_link functionality hasn't been implemented correctly (or at all) for SLURMRunner.

A 404 suggests you are successfully connecting to a web server but the page isn't found. Out of interest what happens if you add /status to the end of the URL it gives you?

If you can share any more information like the links it is giving you that would help troubleshoot this.