`config.yaml` not always available in `md_outputs` branch

[ocaisa]

I've been working on a library of snippets for the HPC Carpentry lessons as we need to make it easy for people to do site-specific customizations of the content. An implementation is in place, and we add something to the config.yaml to pick up the customization.

We need to read this configuration file while building the lesson but the location of the configuration file relative to the content is currently not very predictable. On a first run, the file is included in thte md_outputs branch, However, on a second run the file is removed. In https://github.com/carpentries-incubator/hpc-intro/blob/main/.github/workflows/sandpaper-main.yaml#L61-L68 we've had to customise the deployment workflow to always force a reset of that branch.

Is there a guaranteed location on all branches in the deployment workflow where the configuration file exists?

[froggleston]

Hi @ocaisa - yes, this is down to the way sandpaper uses orphaned branches to build the content, thus "losing" access to the wider lesson files because they're assumed not to be required once the config is loaded into the this_metadata object, learner/instructor globals, etc.

As for a reliable relative location of the config, this isn't easy with the current workflows. It might be easier with the new Docker workflows that will very shortly be available to maintainers.

In these new workflows, a Docker container is used to build the lesson where the lesson is mounted as a volume inside the container under /home/rstudio/lesson - your config would end up being /home/rstudio/lesson/config.yaml. This might work?

As an aside, there is no way to force a reset on each automatic trigger of the ci_deploy(). I can add something to the new build workflow that would allow you to set a repo variable (e.g. vars.ALWAYS_RESET) so it would always reset the build? Might that work?

[froggleston]

This related issue highlights an additional "more correct" way of handling this - by formally pulling in a params or local object into the lesson metadata itself and using that to populate the child markdown snippets. I think this is the way to go, so will try to devote some time to it over the next few weeks.

[ocaisa]

For CI deployment the reset environment variable could work for us, but it will be a little involved to explain to forks (just forking won't be enough, you'll need to set the variable).

The architecture that we have right now is a little complicated to allow us to squeeze into how the sandpaper workflow works. We place customizations in a folder under episodes/files as that was the only way that we could get the folder to be propagated to the orphaned branch.

Under episodes/files/customizations we have folders for each system, each can contain a configuration file (_config_options.yml, which can be empty) and (optionally) a snippets directory. There is a primary customisation (maintained by HPC Carpentry) and a secondary customisation, the idea being that someone only needs to customise the things they actually care about. This secondary configuration is picked up by either an entry in the main lesson config.yaml or by an environment variable. The envvar approach allows us to completely avoid merge conflicts with forks on the main config.yaml, making it easier for forks to stay up to date.

My reason to explain all this is that we actually require very little from the main lesson config.yaml, it only provides the location of the primary and secondary config files so if local was available that would be plenty for us, all the other complexity is hidden away.