Runner client connection error

[crackedupcorson]

Hi

I'm trying to set up the actions-runner-controller, on an EKS cluster, version v1.19.
We're using Vault and ExternalSecretsOperator to pass through the necessary secret values to run organizational-level runners. This is then managed in Argo (v2.0.5) - we have an argo definition for the ExternalSecret and one for the actions-runner-controller. The runner is in our own ECR to avoid hitting docker rate limits.

I have tried both the PAT and GitHub App approaches, and was able to verify successfully that the values existed in the controller-manager secret, including the pem for the GitHub app I was trying to use. When using GH App, the PEM is generated automatically so I'm not worried about the format here.
Argo is able to standup the necessary resources, apart from the RunnerDeployment, the runners failing with this error

Error: Client creation failed. authentication failed: using private key of size 0 (...): could not parse private key: Invalid Key: Key must be PEM encoded PKCS1 or PKCS8 private key

The bit that I'm not understanding is why this error message appears if I'm using the PAT token?
I saw this line in the deployment.yaml and I wonder if it's valid? Should it not be optional?
https://github.com/actions-runner-controller/actions-runner-controller/blob/master/charts/actions-runner-controller/templates/deployment.yaml#L86

When I use the GH app approach, I can verify that the key exists in the necessary secret, but I'm guessing that it can't translate the secret encoding from an ExternalSecret (I can't find anything in pod logs or other kube resources to back this up though).
Sorry about the indentation below, I can't get it to format

Runner Deployment

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
name: actions-runner-deployment
namespace: actions-runner-system
#Defines the RunnerDeployment spec
spec:
template:
#Defines the Runner spec
spec:
labels:
- custom-dubsre-runner
serviceAccountName: actions-runner-service-account
securityContext:
fsGroup: 1000
organization: redacted
image: redacted ECR
imagePullPolicy: Always
dockerdWithinRunnerContainer: true
resources:
limits:
cpu: 0.8
memory: 528Mi
requests:
cpu: 0.4
memory: 256Mi
dockerdContainerResources:
limits:
cpu: 0.8
memory: 600Mi
requests:
cpu: 0.4
memory: 300Mi

Values for Helm Chart (managed with Argo)

actions-runner-controller:
githubEnterpriseServerURL: redacted
name: actions-runner-controller
syncPeriod: 5m
githubAPICacheDuration: 30s
logLevel: debug
podLabels:
some-key: some-value
podAnnotations:
prometheus.io/scrape: 'true'
prometheus.io/port: 'redacted'
prometheus.io/path: '/app/management/prometheus'
serviceAccount:
create: true
name: actions-runner-service-account
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::redacted:role/redacted
authSecret:
create: true
name: "controller-manager"
env:
- name: GITHUB_TOKEN
valueFrom:
secretKeyRef:
name: controller-manager
key: github_token
- name: GITHUB_APP_PRIVATE_KEY
value: /etc/actions-runner-controller/github_app_private_key
service:
type: ClusterIP
port: 443
metrics:
serviceMonitor: false
port: 8443
proxy:
enabled: true
image:
repository: quay.io/brancz/kube-rbac-proxy
tag: v0.10.0
image:
repository: summerwind/actions-runner-controller
DinDSidecarRepositoryAndTag: "docker:dind"
pullPolicy: IfNotPresent
imagePullSecrets:
- name: controller-manager
scope:
# If true, the controller will only watch custom resources in a single namespace
singleNamespace: true
# If scope.singleNamespace=true, the controller will only watch custom resources in this namespace
# The default value is "", which means the namespace of the controller
watchNamespace: ""

Here's our ExternalSecret definition, which I can change to the GH app if needed.

apiVersion: "kubernetes-client.io/v1"
kind: ExternalSecret
metadata:
name: controller-manager
spec:
backendType: vault
data:
- name: github_token
key: redacted/services/github-actions
property: github_token

[crackedupcorson]

I've figured this out.

• I had set the authSecret create to true by accident (when I had it set to false previously), so it was recreating the secret
• I had included the wrong ECR image in the deployment reference for the runner