Adopting the Carpentries Workbench...and reducing divergent forks!

[ocaisa]

At some point (soon) we will need to adopt the Carpentries Workbench. We've discussed trying to reduce our usage of the snippet libraries as they have proved cumbersome to maintain.

One idea that has been floated is to provide example outputs for different schedulers (probably just SLURM and PBS for now) and use tabs to switch between them (one could probably have a config setting that triggers which is the default?). Tab support in the workbench is a WIP.

We can also use child documents as "hooks" for each episode to allow people to have their own fork without ever having to worry about git conflicts. This would allow people to include arbitrary site specific content on a per-episode basis.

[reid-a]

Some references to other work:

There's an issue about templates for snippets that came up at the May 16 HPCC meeting, at the end of which @trhallam (from EPCC) mentions that they have versions of HPC Intro has been ported to the workbench, with snippets. Whether we want to do exactly this, or something like it, or something else remains an open question.

There are a bunch of github repos in the EPCCed project on GitHub, the most recent HPC Intro reflects their Workbench port.

[trhallam]

Hi, thanks for looping me in.

I have experimented with and delivered this carpentry using the new workbench template a few times now (which you have linkted to). FWIW, I don't think it would be easy to teach this set of lessons without the customisation to the location/infrastructure being used for instruction. Thus a slurm or pbs tab wouldn't be helpful, as student at this level expect the examples to work verbatim.

As an example, I taught this class this week using Archer2, which has specific flags and values for the Slurm commands such as the partition names, the qos name and so forth. Archer2 doesn't need you specify the account/project or reservation but a lot of other systems do. Another example, is that different systems have different levels/approaches to authentication, thus connecting to the system is not the same for every cluster.

I don't necessarily think that the current snippets are the best approach, but this ability to customise the carpentry to the teaching system is very important in my opinion.

I have some familiarity with the workbench now, so happy to contribute more if needed.

[trhallam]

Tab support in the workbench is a WIP.

Tabs are now working and I make use of them, particularly when discussing different local systems (mac/linux/windows).

[astroDimitrios]

Hello, author of tabs in the workbench here. I am very interested in snippets/the issue of customisation of commands, setup instructions etc per institution for the Fortran course I'm looking at. I notice in sandpaper issue 81 there is mention of a possible Lua way to add in some customisation I might look at in the next couple of months.

[jlchang]

Hi there, new to HPC carpentries and very interested in a Workbench version of the HPC intro!

For the transition to Workbench, is the next step to try and incorporate UCL snippets into a fork of the Workbench-transitioned EPCCed HPC intro lesson and evaluate whether trhallam's solution (child documents for snippets plus a YAML file for loading snippet into each Rmd) will work and whether the end result seems feasible to current maintainers/instructors? @reid-a @ocaisa @cdkharris @trhallam - If a test of multi-institution snippets sounds like the correct next step, I should have time in October to give it a try if it isn't already on someone's plate.

[astroDimitrios]

@jlchang Perhaps this is something to discuss at the next HPC Carpentry meeting? I have been looking at some work others have started porting the Workbench over to Quarto which has built in support for includes, variables, and conditional content based on metadata. I think this would cover everything in @trhallam's solution.

[trhallam]

Where does Quarto fit into the Workbench ecosystem? Is it an enhancement or replacement for the current Workbench packages?

[astroDimitrios]

@trhallam (I think, currently exploring this based off the work of others in the community) It would combine the custom callout lua from sandpaper and the styling from Varnish into a Quarto extension - so it would be a Workbench (Quarto) version that doesn't rely on/replaces sandpaper, pkgdown, and varnish but would gives us access to all the features present in Quarto.

[jlchang]

@astroDimitrios discussing Quarto vs current Workbench at this week's HPC carpentries meeting and planning next steps sounds great! Which HPC Carpentries meeting do you usually attend (meeting 1 at 13:00 Central European Time or meeting 2 at 17:00 Eastern US Time)? Getting folks interested in a Workbench version of HPC intro onto the same call would be ideal.

[ocaisa]

For my own reference, a kind-of-summary of the status of the Carpentries Workbench and Quarto: UCL-ARC/carpentries-quarto#43

[astroDimitrios]

@jlchang I normally attend meeting 1 (meeting 2 is 10 pm for me so I could, but would need chocolate to stay awake!).

[reid-a]

@astroDimitrios discussing Quarto vs current Workbench at this week's HPC carpentries meeting and planning next steps sounds great! Which HPC Carpentries meeting do you usually attend (meeting 1 at 13:00 Central European Time or meeting 2 at 17:00 Eastern US Time)? Getting folks interested in a Workbench version of HPC intro onto the same call would be ideal.

I will add it to the agenda, it should go out later today.

[jlchang]

@astroDimitrios I'll plan to attend meeting 1 (I'm an early bird so not a problem!) and I'll add the specific meeting to my mention of interest in discussing Workbench'd HPC shell lesson in #hpc-carpentry.

[ocaisa]

For better or worse, we've at least made a choice here. We now use a custom R configuration setup using child documents. The R program doing the heavy lifting is at https://github.com/carpentries-incubator/hpc-intro/blob/main/episodes/load_config.R and configuration value referencing and snippets now look like:

hpc-intro/episodes/13-scheduler.Rmd

Lines 116 to 122 in 7c78ff5

	```bash
	`r config$remote$prompt` `r config$sched$submit$name` `r config$sched$submit$options` example-job.sh
	```
	```{r, echo=FALSE, results='asis'}
	snippets('scheduler/basic-job-script.Rmd')
	```