deploying for lightcone workshop

[minrk]

belatedly writing up tasks for support of the lightcone workshop next week, most of which are done:

Tasks

• deploy cloud build (hub users have access)
• deploy dask gateway
• add nodes for dask
• add lightcone-cli, dask gateway, jupyter ssh
• up baseline reservations so initial launch is quicker
• test scaling users and dask clusters

Currently everything is deployed and I'm testing scaling. I've assigned the dask workers to their own node pool to keep space reserved for users, largely because the workers use a substantially different cpu:memory ratio for scheduling, and it would be wasteful to put them on the cheap e2-highmem nodes we use for users.

Things learned during testing

• increase capacity on the pd-balanced disk is required for satisfactory performance of the NFS server (thanks @yuvipanda!).
• us-central1 doesn't have reliable n4 capacity
• dask gateway adds user: prefix to usernames that contain a colon, which our github users do (I wish this was avoidable, but it is not with CILogonAuthenticator until we have [CILogon] support callable username_claim jupyterhub/oauthenticator#823), so needed to handle that in resolving usernames properly for their home directories

At first, I used the newer n4 node flavor as the current default node flavor, but scaling regularly failed because the us-central1 data center doesn't have enough capacity (wild, since we're talking about just 16 cpus at a time). So I've also added an n2 and n4d pool as a fallback, which seems to launch much more reliably. So if any n4, n2, or n4d nodes are available, I think it should be able to scale on demand.

Still to do ahead of the workshop / during (Monday)

• get attendee usernames so they can access the hub

After the workshop

• scale down dask node pools
• set autoscale minimums back to 0
• possibly disable dask-gateway altogether?
• remove access to cloud build?
• investigate moving nfs server to a hyperdisk, so we can allocate IOPS capacity without also increasing disk capacity. This requires switching our core nodes from e2 to n4 (a 30% price increase, and apparent capacity issues). It also probably requires planned downtime to migrate the data to a new disk (shouldn't be too tough with rsync).