Avoid server restarting when disconnecting from browser

[sebastien-rettie]

To what extent are the coffea-casa images dependent on one staying “connected” via the web browser? E.g. if I start a process/command in the terminal from a web browser, I wonder how long I can stay away/disconnected from the web browser before the process is killed/shutdown? I’ve had both cases on the UChicago cluster, where e.g. I close my browser, reopen https://coffea-dev.af.uchicago.edu/hub/login and the process is still running, but also cases where I log back in and it asks me to login/restart the server, so I’m wondering at what point this actually happens?

Chatting with @oshadura the duration should be 2 weeks, but a few reports seem to indicate this is not the case currently (at least on the UChicago cluster).

[marcus-vgr]

Hi, I wanted to ping this again.
On friday I launched the UChicago server, put some jobs to run, and it was working just fine while I was still using my computer. On the next day when I opened my computer again (although didn't close the cluster-tab), I saw the dashboard has been disconnected in the meantime. So of course the job didn't run till the end. Is there anything we are missing on our side to assure the job will continue running?

Tagging @oshadura

[oshadura]

@fengpinghu could you please take a look on this issue?

Honestly it should not be an issue and I checked with @jthiltges and @clundst and restarting flux should also not trigger shutting down server.

From zero-tojupyterhub developpers they suggest to check "culling" setup https://z2jh.jupyter.org/en/latest/jupyterhub/customizing/user-management.html

[fengpinghu]

Indeed, we have the default culling setup enabled for the Coffea-Casa instances running at UC AF. This means that if a notebook kernel remains idle for approximately one hour, the server will be stopped. While the in-memory session will be lost, notebooks are automatically saved at regular intervals (checkpoints). You should be able to recover your last saved state when you log in again.

[sebastien-rettie]

Thanks for following up @fengpinghu! We also have several people using terminals instead of notebooks; would it be possible to have such a recovery mechanism for terminals too? (I'll let @marcus-vgr comment on if this is the use-case he's thinking of)

[fengpinghu]

Thanks @sebastien-rettie for the clarification.
Checkpoints are only available for notebooks. However, we do have login nodes that you can access via SSH, which might be a good alternative for enabling session recovery.

[marcus-vgr]

Hi @fengpinghu , sorry I am a bit confused. We connect to UChicago via https://coffea-dev.af.uchicago.edu and run a "simple" python script in the terminal. It however can take many hours (perhaps ~day) to finish when we have to run over many datasets / variations. From what I understood, if the server disconnects after some idle time, the python script is killed. How can we make sure the job continues running even though we are no longer connected? Would we need to setup the job via e.g. HTCondor? If yes, would we need some extra configuration to assure the dask cluster will be used properly?

[fengpinghu]

Hi @marcus-vgr, From your description, it sounds like you'd like to keep the Dask cluster alive and run a Python script from the terminal that interacts with it. In that case, the best option would be to extend the notebook server timeout, even if it appears idle. While we don’t want it to run indefinitely, would extending it to 24 hours work for your use case?

[marcus-vgr]

Hi @fengpinghu , yes, I think this would work perfectly! Is this something configured globally by you folks, or can we users modify it for our personal user-cases ?

[fengpinghu]

Hi @marcus-vgr , great. It's already configured globally. You don't need to do anything. Let us know if you encounter any issues. Thanks!

[marcus-vgr]

Hi @fengpinghu , I gave a try again and checked the cluster after being ~3hours disconnected from internet. The python job got frozen, but I do see it using the ps -fA command, although with 99% CPU usage. (gonzalem 2750 136 99 03:47 pts/0 08:44:24 python processEJNT.py -c configs/boosted_example_config.yaml)

Am I missing something?

[fengpinghu]

Hi @marcus-vgr ,

It looks like the notebook is still running and accessible after ~3 hours of idling, which confirms that our configuration change addressed your original concern.

I noticed that there aren’t any Dask workers running for your processing. Are you executing all the processing within the notebook server? How long is it expected to take? This feels more like a Coffea framework question rather than a system-related issue. Perhaps @oshadura could provide some guidance?

[oshadura]

@marcus-vgr do you have somewhere an example of code you are running (e.g. processEJNT.py)?

[marcus-vgr]

Hi @oshadura and @fengpinghu . Perhaps you can access /afs/cern.ch/user/m/mvgonzal/public/processEJNT.py or https://gitlab.cern.ch/atlas-physics/HDBS/DiHiggs/bbbb/bbbbarista/-/blob/master/bbbbarista/processEJNT.py?ref_type=heads ?

The thing is: the whole processing (see L450) before starting the dask execution (L475) takes a very long time if we have too many datasets/systematics. If I have some idle time, the job continues running (although the terminal gets frozen), but it doesn't seem to start the dask execution after the processing ends. I checked that after some ~3hours idle time, disconnected from the internet, I could see the job running via ps -fA, but the dask dashboard has been disconnected. So I think this might be the issue.

PS: the code works perfectly fine when running without these long idle time (i.e., when having less datasets/systematics)

PS2: everything is executed in the terminal with a simple python processEJNT.py --args command

[alexander-held]

Hi @marcus-vgr, how long is "a very long time" for the graph creation? This should hopefully not be more than a few minutes at most. In case your graph becomes extremely complex I could see some startup time for the compute being required as well. Can you split your workflow by (groups of) samples? This would make the graph simpler and I assume you treat those independently anyway.

[marcus-vgr]

Hi @alexander-held, it is O(minutes) for just one sample with several systematics (or several samples with few systs), but once we put to run everything, I estimate almost a 20hours processing. So I guess the solution should be really splitting the whole workflow