gmsaWebhooks report "bad certificate" when using cert-manager.

[mark-bixler]

I successfully have cert-manager running following all the resources defined here:

https://github.com/kubernetes-sigs/windows-gmsa/blob/master/charts/gmsa/templates/issuer.yaml

The mutating & validating webhooks are referencing the proper CA as outlined in the templates as well.

Proper value for the secretName as well.

The logs for the gmsaWebhook continue to show:

│ 2024/10/01 04:28:21 http: TLS handshake error from 10.x.x.x:45168: remote error: tls: bad certificate

Completely stumped as to what is missed here.

I've port forwarded gmsaWebhook Pod and tested out SSL and get the errors:

depth=0 CN=gmsa-webhook.gmsa-webhook.svc
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=gmsa-webhook.gmsa-webhook.svc
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN=gmsa-webhook.gmsa-webhook.svc
verify return:1

ENVIRONMENT:
Image: 'registry.k8s.io/gmsa-webhook/k8s-gmsa-webhook'
Tag: 'v0.8.0'
Latest Chart Template versions.

For further context, we have a working implementation when using the gmsa-webhook-cert.sh instead of cert-manager.

[jsturtevant]

I've upgraded the cert-manager version in #154, could you try with that?

I don't think this is a webhook issue but might be related to either cert-manager or a mis configuration of the CRD's. Can you check the logs of cert-manager?

[mark-bixler]

Nothing in the logs of cert manager. I can try and bump to v0.9.0 but I'm not optimistic at all.

Is there any other missing piece for the CA or cert that isn't shown in the issuers.yaml?

Is it correct to reference the Self-Signed CA we create with ClusterIssuer in our mutating & validating webhooks and then reference the TLS cert as the secret name for the deployment?

You mean the CRD's for Cert-Manager? The ones for gmsaWebhook (afaik) are the just for the credentialSpec.

[jsturtevant]

I have a working deployment using k8s 1.29 and cert-manager. The deployment looks like:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: windows-gmsa-dev
    meta.helm.sh/release-namespace: windows-gmsa-dev
  creationTimestamp: "2024-10-01T16:04:09Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    chart: gmsa-0.9.0
  name: windows-gmsa-dev
  namespace: windows-gmsa-dev
  resourceVersion: "1085"
  uid: 42aaad72-edab-4b3f-9f41-b9275d99e673
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: windows-gmsa-dev
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: windows-gmsa-dev
    spec:
      containers:
      - args:
        - --cert-reload=false
        env:
        - name: TLS_KEY
          value: /tls/key
        - name: TLS_CRT
          value: /tls/crt
        - name: HTTPS_PORT
          value: "443"
        - name: BURST
          value: "50"
        - name: QPS
          value: "30"
        image: sigwindowstools/k8s-gmsa-webhook:v0.9.0-1-gf4f11fc
        imagePullPolicy: IfNotPresent
        name: windows-gmsa-dev
        ports:
        - containerPort: 443
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 443
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tls
          name: tls
          readOnly: true
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      os:
        name: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: windows-gmsa-dev
      serviceAccountName: windows-gmsa-dev
      terminationGracePeriodSeconds: 30
      volumes:
      - name: tls
        secret:
          defaultMode: 420
          items:
          - key: tls.key
            path: key
          - key: tls.crt
            path: crt
          secretName: gmsa-server-cert

with two secrets created by cert-manager:

k get secrets
NAME                                     TYPE                 DATA   AGE
gmsa-server-cert                         kubernetes.io/tls    3      8m45s
sh.helm.release.v1.windows-gmsa-dev.v1   helm.sh/release.v1   1      8m50s
windows-gmsa-dev-root-ca                 kubernetes.io/tls    3      8m50s

[mark-bixler]

And your mutating & validating webhooks referencing the root CA?

[jsturtevant]

cert manager is injecting the cert:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from: windows-gmsa-dev/windows-gmsa-dev-ca

which is wired to the root-ca cert:

k get certificates
NAME                  READY   SECRET                     AGE
windows-gmsa-dev      True    gmsa-server-cert           72m
windows-gmsa-dev-ca   True    windows-gmsa-dev-root-ca   72m

[mark-bixler]

Hmmm, that's exactly what I'm doing as well.

[jsturtevant]

Can you check the statuses of the various cert-manager CRD's? Maybe something went wrong during the issuing?

[mark-bixler]

CRD's all look good. For our "working" implementation w/o cert-manager our CN is "Kubernetes" and here it's the gmsa-webhook-ca. It feels as if the chain created in the issuers.yaml just doesn't work with the deployment.

[jsturtevant]

Did you install the webhook and cert-manager via the helm chart or was this manual install?

[mark-bixler]

webhook via helm & cert-manager via flux

…

On Tue, Oct 1, 2024 at 10:20 AM James Sturtevant ***@***.***> wrote: Did you install the webhook and cert-manager via the helm chart or was this manual install? — Reply to this email directly, view it on GitHub <#153 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKRH7GKY7KYRTB3C2Q32RWTZZL7XXAVCNFSM6AAAAABPEWRFTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBWHE4TENBZHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[mark-bixler]

It must be how the service / pod reads the secret. I've verified the certificate chain is valid.

[jsturtevant]

webhook via helm & cert-manager via flux

my guess is there is something not quite hooked up right between the two components. Is it possible to try installing certmanager with the helm chart to see if that works and then do a compare of the all the manifests to find what is missing?

[mark-bixler]

It's not cert manager. It works for all other services, just not the gmsaWebhook.

I'm giving up.

[jsturtevant]

Understand its not cert-manager. We do have CI working (plus I ran it locally) with cert-manager so I know it works. I am suggesting something between the configuration of webhook deployment and cert-manager isn't correct in your env. Trying to suggest a few ways to debug, its hard to diagnose on your configuration on this end without dumps of the crds/deployments. Maybe restarting the webhooks will pick up the correct cert?

[mark-bixler]

The webhooks get the correct cert, I can verify & decode that against the root CA. And if I setup a test pod with the same volume mounts as the gmsaWebhook I can see the proper values there from the secret. Unfortunately I'm not able to exec into the webhook pods themselves to verify. They seem like a black box. Is there a dockerfile for those to see what's going on?