Runnerset and PVC

[ViktorLindgren95]

When using RunnerSet with Persistent Storage, everytime we scale up or scale down aswell as use ephemeral runnerset the storage is lost between jobs. The statefulset is recreated together with the job pod causing the Pvc to be removed and another bounding has to happen. This makes it take about 5 minutes for a new runner to start every time. We have resorted to using Persistent runners for now but would love to be able to use Ephemeral runnerSet together with docker image cache etc like its documented in the readme.

Structure looks like

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: cached-runnerset-a
spec:
  ephemeral: true
  group: gaas
  enterprise: removed
  labels:
    - gaas-cached
    - cached-runnerset-a
  selector:
    matchLabels:
      app: cached-runnerset-a
  serviceName: pipeline
  template:
    metadata:
      labels:
        app: cached-runnerset-a
    spec:
      serviceAccountName: arc-runnerset
      containers:
        - name: runner
          env:
            - name: HTTP_PROXY
              value: "removed"
            - name: HTTPS_PROXY
              value: "removed"
            - name: NO_PROXY
              value: "removed"
            - name: DISABLE_RUNNER_UPDATE
              value: "true"
          resources:
            limits:
              cpu: 500m
              memory: "1Gi"
            requests:
              cpu: 100m
              memory: "1Gi"
        - name: docker
          env:
            - name: HTTP_PROXY
              value: "removed"
            - name: HTTPS_PROXY
              value: "removed"
            - name: NO_PROXY
              value: "removed"
          resources:
            limits:
              cpu: 500m
              memory: "4Gi"
            requests:
              cpu: 100m
              memory: "4Gi"
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /var/lib/docker
              name: var-lib-docker-a
  volumeClaimTemplates:
    - metadata:
        name: var-lib-docker-a
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: arc-var-lib-docker

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: arc-var-lib-docker
  labels:
    content: arc-var-lib-docker
provisioner: csi.vsphere.vmware.com
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Is our configuration wrong? Is this how its supposed to work that the PVC is removed and our provisioning is just too slow for this configuration etc. I would love some answers if possible

[mumoshu]

The statefulset is recreated together with the job pod causing the Pvc to be removed and another bounding has to happen.

AFAIK K8s doesn't allow dynamically rebounding a bound PVC to another PV or remount bound PVC/PV to another pod. It needs to unmount and unbound before the PV is reused by another pod. The same for runner pods. Note that it's not PVCs to be reused. PVs are reused, by newly created PVCs being bound to existing eligible PVs.

This makes it take about 5 minutes for a new runner to start every time.

I think "every time" is not correct. Due to how it works, a new runnerset statefulset's PVC needs a already "Available" PV for reuse. If there's no such PV, the K8s pvc controller will dynamically create a PV and it might take 5 minutes for that specific pod. This happens when your pv provisioner and the k8s control-plane is slow enough so that it can't unbound the PV from the old runner pod before the new statefulset gets created.

After a few trials, you might see the number of PVs will max out at certain number, depending on how many new runner statefulsets/pods can be concurrently created at a time(And it highly depends on your workflows, not ARC or K8s or GitHub Actions). After that, every newly created statefulset PVC will see at least one available PV to be reused, hence the "5m" delay you saw disappear. Would you mind confirming?

[ViktorLindgren95]

What is the theoretical time of a runnerset with a persistent volume to start ? Without it's about 15 seconds from non existent to serving job

[roshvin]

we uses volumeClaimTemplates and it will attach in seconds to the runner .

[ViktorLindgren95]

@roshvin Could you go step by step what happens in your cluster? from 0 to serving job ?

[ViktorLindgren95]

So after running some stresstests during the day, the delay never went down. It could be the cluster im working on (onprem) has something weird with its provisioning but it stills takes about 5 minutes to bind the PVC to a PV, I could see PVs i had created earlier that were days old.

I will investigate further

[mumoshu]

Just to be clear i don't have to create a Pv, the system is supposed to create both PVs and PVCs?

Generally speaking, yes. It's a vanilla Kubernetes feature called "Dynamic Volume Provisioning" https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/.
You can still manually provision PVs if you disable the dynamic provisioning. But I think that's not what everyone wants to do.

[ViktorLindgren95]

The problem seem to be the CSI driver my enterprise onprem is running has some kind of bug, as it works on my local cluster.

Thanks for a quick answer

[ViktorLindgren95]

@mumoshu I have double confirmed now that it was our CSI driver / API server that was the issue, as our mirror cluster worked perfectly. Thank you for your time

[shinji62]

@mumoshu something related to this question, I would like to understand how ARC understand which PV to use ? As the PVC is deleted when the pod is deleted and the volume become available, how does ARC know which PV to use.

Second things, the PV / PVC created trough ARC even with the retainPolicy: Delete are not deleted from the kubernetes and even deleting them manually from kubernetes will not delete them from AWS (using csi ebs), so I am wondering if ARC is doing anything to create those PV/PVC as well.

[mumoshu]

Hey!

how does ARC know which PV to use.

ARC creates PVC via RunnerSet('s embedded StatefulSet spec). It's the underlying K8s statefulset API and the controller that maps the PVC to an existing PV. It seems like a PV with the same storageClass is eligible for reuse, but you may better refer to the k8s documentation about the full explanation.

Second things, the PV / PVC created trough ARC even with the retainPolicy: Delete are not deleted from the kubernetes and even deleting them manually from kubernetes will not delete them from AWS

I believe that's how StatefulSet(which is the foundation of ARC's RunnerSet) works today.

There's an alpha Kubernetes feature that let StatefulSet be able to actually delete the PVs https://kubernetes.io/blog/2021/12/16/kubernetes-1-23-statefulset-pvc-auto-deletion/

[zezaeoh]

Second things, the PV / PVC created trough ARC even with the retainPolicy: Delete are not deleted from the kubernetes and even deleting them manually from kubernetes will not delete them from AWS

For me, this is because of external-provisioner's bug :)
kubernetes-csi/external-provisioner#796

[harshaisgud]

@mumoshu Sorry to bring up an old converstaion. Please correct me if I am misunderstanding this but the feature that you pointed out(https://kubernetes.io/blog/2021/12/16/kubernetes-1-23-statefulset-pvc-auto-deletion/) is in relation to PVCs and not PVs. As @shinji62 pointed out. Even though the Requesting PVCs are deleted the provisioned PVs still hang around even with retain policy set to Delete. I would be happy to open an other issue and help with debugging . I suspect the finalizer kubernetes.io/pv-protection is the reason the PVs are retained.

[verult]

I believe that the deletion of PVC once the pod is deleted is actually not standard StatefulSet behavior. When I create a simple StatefulSet and then delete a pod or the entire StatefulSet, all created PVCs are retained. This is assuming that the alpha feature mentioned above for StatefulSet PVC auto deletion is disabled.

[verult]

As someone who has worked on k8s storage internals, I wanted to share the perspective that the PVC deletion and PVC-PV unbind logic is at odds with how the k8s storage system is intended to work. The RunnerSet PVC logic has a few issues:

1. The intention of the core k8s PV controller is that this is the only controller in a k8s cluster that handles PVC-PV binding. There may be race conditions in the PVC/PV lifecycle if components interfere.
2. After deleting the PVC, there is no way to delete the underlying storage from k8s. You can still delete it by talking to the storage system (GCE) and manually deleting the PV, but it's manual and error prone. For volumes using a StorageClass with reclaimPolicy: Delete, storage deletion is designed to be triggered by PVC deletion, not PV deletion. Which brings up the last issue...
3. StorageClass has a field called reclaimPolicy, which controls what happens to the PV after a PVC is deleted. Possible values are Delete and Retain. By having a separate controller unbind the PVC-PV pair, the reclaimPolicy is violated if it's set to Delete.

In order to reuse cache stored in a volume, the PVC can be left in place. A new StatefulSet runner replica with the same index as the PVC (as suffixes in their names) will automatically mount the volume the PVC references.

Curious what folks think, and please correct me if there's anything I misunderstood. I'm also curious to hear more about the original intention of the PVC deletion and PVC-PV unbind logic, and happy to help brainstorm other ways to solve the original problem.

[remover]

so your suggestion would be not to try to delete the PVs manually, right? if that's the case then it probably makes the docker layer caching strategy difficult to use cos in my testing it seems like the number of PVs will continue to grow.

i think this happens cos all PVs can be already bound at peak times, causing new ones to be created. if that's true then maybe there should be some natural upper bound on the number of PVs that get created but just curious for your thoughts...

[sagar-bud]

Hello,

I have registered the GitHub action runner in GKE cluster at organisation level. How we can use same docker cache PV with multiple runners pod. When I runner pipeline in Multiple repo within organisation, how runners pod use same PV to store docker cache.

I understood, if one runner pod is running at a time, it will use existing PV to store docker cache but when multiple runners(more than one or two) pods are running at same time. how runner pods will use same PV.

[sagar-bud]

@mumoshu

Do you have any idea? How we can store docker caching when multiple pipeline pods are running?

[shaojun-tes]

+1

[sagar-bud]

Hello @shaojun-tes

Please refer the below URL for the above.
#1362

[shaojun-tes]

@sagar-bud got it, thx very much!

[zakariais]

Hi
I have a questions regarding this
we are trying to implement nvme storage with github actions.
We tried with the RunnerDeployment but the ARC doesn't clean up the nvme disk after the actions finishes ( we used volume and volumeMounts of k8s, there is an issue already opened for it)
I want to know if we move to runnerSet and use the localPV to provisio PV and the pvc claim, if we schedule multiple pods on the same node point to the same PV, it won't work right as volume is binded to one pod and other pod will keep on pending?
How can we use disk nvme with multiple pods to same node and utilize scale up and down with github action workers?
@mumoshu