Skip to content

Conversation

@froggleston
Copy link
Member

@froggleston froggleston commented Apr 28, 2025

This PR is a fairly consequential update to the sandpaper workflows. The main provisions are:

  • When Workbench package releases happen (i.e. a sandpaper, pegboard or varnish release), these will be built into an official docker container by the Core Team. The new workflows will use this latest docker image so users will no longer have to merge in PRs to update the Workbench for their lessons. They will get them automatically!
  • Locally pinned packages will work now in Github Actions, which they do not currently.
  • For lessons that use renv, there are now two workflows instead of one. One workflow checks for packages and raises a PR if required, and a separate renv cache application workflow that allows users to choose when they want to apply an updated cache and store it as an artifact. This means that build and deployment will use the same renv cache until the user decides to update, reducing maintenance time and support from workbench devs, faster builds, and consistent environments until a user chooses to update. PRs will still be raised on a schedule as normal when package updates are detected.
  • Dockerised version of the pr_receive workflow, removing the need for the SANDPAPER_TOKEN, making lesson setup easier on GitHub
  • An updated update-cache workflow, removing the need for the SANDPAPER_TOKEN as per Remove the need for a custom SANDPAPER_WORKFLOW PAT #646 (thanks @Bisaloo!)
  • Dockerised version of the build_deploy workflow, removing the requirement of sandpaper-main.yml and sandpaper-version.txt. This means the workflow will always renv::restore the same cached renv environment for each build rather than updating packages whether intended or not via remotes::install() and sandpaper::manage_deps().
  • Extra options available to the package update check workflow to allow skipping renv::restore, effectively building a new lockfile with completely new set of updated packages, including bioconductor lessons, This provides an easy way to update lesson dependencies when packages get old, or new versions of Bioconductor and/or R are released.
  • Dev options to (a) disable any package updates, which helps testing, and (b) to turn off generating the temporary renv cache when raising the package update PR lesson build tests.

Testing needs to be done, and then a formal mechanism to switch to the new dockerised versions, presumably by replacing the existing workflow files, and using the existing update-workflows mechanism to propagate through user's repos.

What left to do:

  • Fix the tests for update_workflows (removal of sandpaper-main etc)
  • Develop and test a reliable way for people to disable existing workflows and move over to these workflows
  • Update repository documentation
  • Update sandpaper-docs documentation (planned)
  • Update workbench-dev documentation (in progress)

Documentation for testing:

Clone froggleston's sandpaper and switch to the docker_workflows branch:

  • cd ~
  • mkdir ~/docker-test
  • cd docker-test
  • git clone https://github.com/froggleston/sandpaper
  • cd sandpaper
  • git switch docker_workflows
  • Fork and clone a lesson
    • Fork swcarpentry/git-novice in the github web interface to <your_github>/git-novice
      • Make sure you uncheck the main branch only option so we get all branches
    • mkdir ~/lessons
    • cd ~/lessons
    • git clone <your_github>/git-novice
  • Change directory to your local lesson you want to update the workflows for, e.g.
    • `cd ~/lessons/git-novice
    • R
  • In R:
    • install.packages("devtools")
    • devtools::install("~/docker-test/sandpaper")
    • sandpaper::update_github_workflows()
    • quit()
  • git add .github/workflows
  • git commit -m "Manual update to docker workflows"
  • git push origin main

This will automatically start the 01 Maintain: Build and Deploy Site workflow. This will work for non-renv lessons. For renv-enabled lessons:

  • As the lesson will have no renv cache, the "01 Maintain: Build and Deploy Site" run that automatically started following the push to main will fail. This is OK!
  • Run "02 Maintain: Check for Updated Packages"

If packages were updated and a PR raised, the subsequent "03 Maintain: Apply Package Cache" and "01 Maintain: Build and Deploy Site" workflows should run automatically.

If no updates were made, run "03 Maintain: Apply Package Cache" manually, and then "01 Maintain: Build and Deploy Site" will run automatically.

@froggleston froggleston self-assigned this Apr 28, 2025
@froggleston froggleston added enhancement New feature or request package cache Related to the {renv} package cache frequency: high indicator that a use-case has a high-frequency in lessons Developer QOL 👷 This feature enhances the developer quality of life labels Apr 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Developer QOL 👷 This feature enhances the developer quality of life enhancement New feature or request frequency: high indicator that a use-case has a high-frequency in lessons package cache Related to the {renv} package cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant