Github Actions running on Scale Set are not able to complete jobs

[mxw-sec]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.10.1

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Deploy Github Runner Controller and Scaleset on EKS.
Create a Job that downloads the codebase from github and then reads a file and updates the readme.

Describe the bug

/usr/bin/tar: XXXXXXXXX Cannot change ownership to uid 0, gid 0: Operation not permitted
/usr/bin/tar: Exiting with failure status due to previous errors
Error: The process '/usr/bin/tar' failed with exit code 2
Error: Error: failed to run script step: command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/a133c0e0-cd61-11ef-9cd1-fdc32e02a729.sh], exit code 1
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.

I have tried use GP2, GP3, and EFS Volumes.

GP2 and GP3 Support the init-container workaround, but I run into other issues, such as the job files are not found in the Job Container.

Describe the expected behavior

Permissions issues do not exist.

Additional Context

# Chart: git@github.com:actions/actions-runner-controller
# git ref: 4357525445b0b77388af4e1f171b5b7bd9b116a4
# Path: charts/gha-runner-scale-set

      githubConfigSecret: pre-defined-secret
      githubConfigUrl: "https://github.com/{ORG}"
      controllerServiceAccount:
        namespace: arc-systems
        # Name must line up with the above chart release name eg `github-actions-scale-set-controller`. Install the above chart and see what SA name it makes.
        name: arc-gha-rs-controller
      minRunners: 1
      maxRunners: 50
      containerMode:
        # Needed, even if the docs say it isn't.
        type: kubernetes
      kubernetesModeServiceAccount:
          annotations:
            # https://github.com/actions/actions-runner-controller/blob/98854ef9c018141d7386657322da351e11029da2/charts/gha-runner-scale-set/tests/values_kubernetes_mode_service_account_annotations.yaml#L4
            eks.amazonaws.com/role-arn: arn:aws:iam::{ACCOUNT}:role/{ROLE}
      kubernetesModeWorkVolumeClaim:
        accessModes: ["ReadWriteOnce"]
        storageClassName: "gp3"
        resources:
          requests:
            storage: 5Gi
        # Here is how your make the runner pods have a custom IAM Role, so they can (eg) contact real resources in you AWS account.
      template:
        spec:
          # We must add an init container to change the ownership of the _work directory
          # https://docs.github.com/en/enterprise-server@3.9/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors#error-access-to-the-path-homerunner_work_tool-is-denied
          initContainers:
          - name: kube-init
            image: ghcr.io/actions/actions-runner:latest
            command: ["sudo", "chown", "-R", "1001:123", "/home/runner/_work"]
            volumeMounts:
            - name: work
              mountPath: /home/runner/_work

          # We have to fully override the containers simply to set our own "image"
          containers:
          - name: runner
            # This image is used as the runner image.
            # Note it cannot be "your image `FROM ubunutu` or similar, it must be based off the one in https://github.com/actions/runner/blob/main/images/Dockerfile
            # Or you can build your own and try and include all the items from that build.
            image: ghcr.io/actions/actions-runner:latest
            command:
            - /home/runner/run.sh
            env:
            - name: ACTIONS_RUNNER_CONTAINER_HOOKS
              value: /home/runner/k8s/index.js
            - name: ACTIONS_RUNNER_POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
              value: "false"
            volumeMounts:
            - name: work
              mountPath: /home/runner/_work
          volumes:
          - name: work
            ephemeral:
              volumeClaimTemplate:
                spec:
                  accessModes: [ "ReadWriteOnce" ]
                  # Critical change here compared to the docs. EKS does not support "local-storage" by default.
                  storageClassName: "gp3"
                  resources:
                    requests:
                      storage: 2Gi




OR 


githubConfigSecret: pre-defined-secret
## maxRunners is the max number of runners the autoscaling runner set will scale up to.
maxRunners: 50
minRunners: 2
githubConfigUrl: "https://github.com/helium10"
containerMode:
  type: "kubernetes"
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "gold"
    resources:
      requests:
        storage: 5Gi
template:
  spec:
    securityContext:
      runAsUser: 1001
      runAsGroup: 0
      fsGroup: 0
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        env:
          - name: ACTIONS_RUNNER_CONTAINER_HOOKS
            value: /home/runner/k8s/index.js
          - name: ACTIONS_RUNNER_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
            value: "true"
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
    volumes:
      - name: work
        ephemeral:
          volumeClaimTemplate:
            spec:
              accessModes: ["ReadWriteOnce"]
              storageClassName: "gold"
              resources:
                requests:
                  storage: 1Gi

Controller Logs

v

Runner Pod Logs

a

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[YodaKV]

More context's need. Please post logs from controller, listener pod, output from kubectl -n XXX get all (replace XXX with your namespace)