incompatible with ARE _desktop_ environment

[davidmonro]

Hi Dale,

more a note than an actual issue - this approach won't work in the ARE VDI desktop environment, because that is itself implemented as a container, and nested containers are not a thing on Gadi.

It should however work fine in the ARE jupyterlab environment.

I don't think there's anything you can do to make it work, but you can probably make it fail more gracefully that it does currently by looking for existing SINGULARITY* environment variables.

There is a kind-of workaround - you can always ssh back to localhost on the node, and run there, since that allows you to escape from the container back into the 'normal' gadi environment (but still contained in the same job).

---

David Monro
Linux Systems Administrator
NCI Australia
The Australian National University
56 Mills Road
Acton, ACT, 2601

[dsroberts]

Hi David

Thanks for the note. Yes, I do know that this doesn't work on the virtual desktop environment. The priority was Gadi command-line/PBS jobs and ARE jupyterlab, which covers the majority of the environment's use case. The failure you're seeing is because it is detecting that its in a container and assuming that the in-container paths exist. More graceful failure would be a good thing though. I'll put a pin in this for now and think about it some more, maybe I can tell your container to mount the squashfs somehow...

Dale

[dsroberts]

Hah... I can! If I add SINGULARITY_OVERLAYIMAGE=/g/data/hh5/public/apps/cms_conda/envs/analysis3-22.10.sqsh to the list of PBS -v arguments, your container will mount my squashfs, and I can use it from within the virtual desktop environment.

[rbeucher]

argh...Was planning to use xp65 with VDI...

[rbeucher]

I can mount the squasfs just fine. It's just the extra-step that might be annoying for the users..

[dsroberts]

You could always sneak it into the module file. The logic could look like:

if PBS_NCI_OOD == desktop_vnc
    setenv SINGULARITY_OVERLAYIMAGE /g/data/hh5/public/apps/cms_conda/envs/analysis3-23.07.sqsh

Conversion to tcl is left as an exercise to the reader (or ChatGPT)

[rbeucher]

OK passing the option to the VDI settings works via PBS -v works.

Now modifying the module file (common_v3) with something like:

### For VDI, we need to set the 
if {$env(PBS_NCI_OOD) eq "desktop_vnc"} {
    set env(SINGULARITY_OVERLAYIMAGE) "/g/data/xp65/public/apps/med_conda/access-med-0.3.sqsh"
}

does not work

[dsroberts]

@rbeucher thats the answer ChatGPT gave me last night, unfortunately that's not how you set an environment variable in a module file.

setenv SINGULARITY_OVERLAYIMAGE /g/data...

Probably best to use an internal module variable for the path. There is probably one already set in the module.
You'll also need to handle the case where PBS_NCI_OOD is not defined. NCIs OpenMPI module has a good example on handling this.

[rbeucher]

Haha. I thought I would give a go at ChatGPT that time.. my bad

[rbeucher]

Yeah no luck. I set the SINGULARITY_IMAGEOVERLAY variable and get:

/g/data/xp65/public/apps/med_conda_scripts/access-med-0.3.d/bin/sed: line 82: /g/data/xp65/public/apps/med_conda/envs/access-med-0.3/bin/sed: No such file or director

Lets see on Monday when you get back. No rush

[dsroberts]

Hmmm... The catch here is that for the setting in the module to work, the module will need to be loaded before the VDI environment launches. Given that the VDI launch page remembers your settings, I think it'll be simple enough to instruct users to add SINGULARITY_OVERLAYIMAGE=... once, just like the Environment module settings in the Jupyterlab launch page.

[rbeucher]

OK. I thought we could avoid that but I reached the same conclusion. Thanks @dsroberts