Support port forwarding in SSHCluster

[jacobtomlinson]

Some users may have SSH access to a machine and want to use SSHCluster with it, but are not able to access ports other than 22 due to firewall restrictions.

It would be nice to be able to forward the scheduler ports locally through the SSH tunnel.

This would require:

• Add kwargs to SSHCluster to specify local port forwards
• Set up port forward in scheduler object after connection is made (docs)
• Set the scheduler address to the locally forwarded port here

Example

>>> from dask.distributed import Client, SSHCluster
>>> cluster = SSHCluster(
...     ["host1", "host2", "host3", "host4"],
...     forward_scheduler_port=8786,
...     forward_dashboard_port=8787
... )
>>> client = Client(cluster)

There is an assumption here that the restrictive firewall rules would not limit worker-scheduler and worker-worker communication.

[mrocklin]

Yeah, I'm in favor. Port forwarding is part of my routine when using Dask, and I always have to look up how to do it on the command line.

…

On Wed, Jun 10, 2020 at 9:13 AM Jacob Tomlinson ***@***.***> wrote: Some users may have SSH access to a machine and want to use SSHCluster with it, but are not able to access ports other than 22 due to firewall restrictions. It would be nice to be able to forward the scheduler ports locally through the SSH tunnel. This would require: - Add kwargs to SSHCluster to specify local port forwards - Set up port forward in scheduler object <https://github.com/dask/distributed/blob/6ee7f899d1bc686ba4910c08509c7f2e5345506d/distributed/deploy/ssh.py#L170> after connection is made (docs <https://asyncssh.readthedocs.io/en/latest/api.html#asyncssh.SSHClientConnection.forward_remote_port> ) - Set the scheduler address to the locally forwarded port here <https://github.com/dask/distributed/blob/6ee7f899d1bc686ba4910c08509c7f2e5345506d/distributed/deploy/ssh.py#L206> *Example* >>> from dask.distributed import Client, SSHCluster>>> cluster = SSHCluster( ... ["host1", "host2", "host3", "host4"], ... forward_scheduler_port=8786, ... forward_dashboard_port=8787 ... )>>> client = Client(cluster) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3881>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTDTZ3FOKD6PADPC4RDRV6WJHANCNFSM4N2RCKSQ> .

[quasiben]

@drabastomek you may be interested in the PR -- I believe the Azure PR: dask/dask-cloudprovider#67 was using some port forwarding tricks

[drabastomek]

Thanks @quasiben! I believe we finally worked out the PF. I will be checking in the changes today.

[moritzaugustin]

is there an estimated date when this will be finally merged via #3895 ?

i would very much like to test the port forwarding feature in my dask setup (tens of servers) where i currently rely on manual port forwarding management

[TomAugspurger]

@moritzaugustin seems like #3895 has some merge conflicts and test failures. I'm guessing @jacobtomlinson would be fine with you picking up that pull request if you're interested.