Use cases for BlueAPI Deployment PVC

[DominicOram]

As a support engineer wanting to manage my deployments I would like to do the following (in order of importance):

• Roll back to a known "good" PVC
• Have BlueAPI make a new PVC pulling a clean main (or a release tag) of my repos, the current PVC should be kept around
• Remotely see if my current PVC has uncommited changes
• Read the code in an old PVC without changing live
• Delete an old PVC
• See when a PVC was made/last used/last edited
• Delete based on some criteria e.g. all that haven't been used for X time

Acceptance Criteria

• All the above are documented on how to do them
• If they are hard or require more than 2 or 3 steps there are issues to make them easier

[fajinyuan]

Hi @DominicOram Thanks for the request. Could you please clarify in this ticket a bit more - what is this PVC for? is it for

• storage for user scripts?
• storage for visit experiment data?
• storage for workspace of application cache?
• or anything else?
  we may need to develop policy for each of these storage type if not already available.
  I am not yet a Cloud software developer, and not sure PVC can or cannot be rolled back or forth independent of pod.

[coretl]

@hyperrealist @gilesknap please can you think about whether we have the same requirements for FastCS / epics-containers dev PVCs

[DominicOram]

Could you please clarify in this ticket a bit more - what is this PVC for?

This is for the PVC that BlueAPI currently uses for deployments i.e. what mounts village-bluesky plan repositories and the python environment that runs them (so dodal + ophyd-async etc.). So I guess from your list "storage for workspace of application cache?" is the closest match.

storage for user scripts?
storage for visit experiment data?

I'm not sure what the D-II architecture is for storing these? At the moment I have assumed experiment data is stored in /dls/ixx/data as previously and I have sidestepped the question of user scripts. If DAQ core have a plan to also store these via PVC then I would probably have some similar requirements I could feedback but I suggest we cross the bridge when we come to it.

[fajinyuan]

You are correct. I was just told by the team the detector data and nexus files are currently not accessed via PVC, it still uses Posix file system directly. I am not aware of this decision, nor for what reasons this had been decision as such.
There still no discussion on how to handle storage for user scripts so far.
So what you asked here is for deployment source codes, thanks for confirming this.

[dan-fernandes]

Of the criteria, the most difficult will be (in no order):

1. Seeing when a PVC was last edited - I think this will be tricky in the general case, but more possible if scoped to the currently mounted PVC. We already have code (not shipped) to tag PVCs with when they were last used. Hopefully this largely fulfils the use case.
2. Auto-deletion - Again, we have an un-shipped feature to auto-delete on pre-defined criteria. We're unable to ship this because we don't have a good solution to alerting staff prior to deletion.

I think the rest are achievable. However the human interaction with them is all quite messy:

• Mounting a specific PVC / creating a new one which pulls from main will be done by changing a field in the Values.yaml file. It can also be overridden in ArgoCD (not recommended).
• Remotely viewing whether a PVC has uncommitted changes can done via the /python_environment endpoint, eg. this response body having edited P48:

{
  "installed_packages": [
    {
      "name": "dls-dodal",
      "version": "https://github.com/DiamondLightSource/dodal.git @main",
      "location": "/app/.venv/lib/python3.11/site-packages &&/workspace/dodal/src &&",
      "is_dirty": false,
      "source": "scratch"
    },
    {
      "name": "htss-rig-bluesky",
      "version": "https://github.com/DiamondLightSource/htss-rig-bluesky.git @foobar",
      "location": "/app/.venv/lib/python3.11/site-packages &&/workspace/htss-rig-bluesky/src &&",
      "is_dirty": true,
      "source": "scratch"
    }
  ],
  "scratch_enabled": true
}

• A PVC's last use is stamped into its Kubernetes annotations. This enabled PVC auto-deletion, but is not very human friendly. You can get the timestamp with a Kubectl command, or by looking at the beamlines' Kubernetes dashbaord or ArgoCD:

Feedback on whether this needs to be rationalised into a more singular experience would be great. Nonetheless they're technically possible.

[Alexj9837]

A link to the PR with the PVC auto deletion + time stamping logic #1514

[gilesknap]

re @coretl question. My view is that for a dev PVC for FastCS or epics-containers IOCs we should keep it very simple and have one per service being 'deved'. However, if we are to unify the approach to creating these PVCs then having all of the above requirements available, yet also make it easy to pick the simple 1-to-1 pattern would also be fine.

[DominicOram]

My view is that for a dev PVC for FastCS or epics-containers IOCs we should keep it very simple and have one per service being 'deved'

This is interesting, how will you deal with the situation where you have a hotfixed IOC and you roll forward to a release but this doesn't work and you have to roll back?

[gilesknap]

My view is that for a dev PVC for FastCS or epics-containers IOCs we should keep it very simple and have one per service being 'deved'

This is interesting, how will you deal with the situation where you have a hotfixed IOC and you roll forward to a release but this doesn't work and you have to roll back?

My expectation is that rolling forward would mean switching off debugging mode and just using the native container without any overlay PVC, thus leaving the hotfix in place. But, if I wanted to juggle two different hotfixes then maybe I do need versioning.