step-issuer failing to connect to step-certificates using JWK

[yggur-au]

step-issuer.values.txt
Hi team,
I am attempting to deploy (via the smallstep Helm chart) an instance of step-issuer on AKS. The cluster is running:

• Kernel version: 1.27.1
• step-certificates: v0.24.2 (also deployed via the smallstep Helm chart)
• step-issuer: v0.7.0

I am receiving the following error when deploying the step-issuer instance:
{"level":"error","ts":"2023-08-15T04:51:40Z","logger":"controllers.StepClusterIssuer","msg":"failed to initialize provisioner","stepclusterissuer":"/ecdsa-aks-step-issuer","error":"error parsing provisioner encrypted key: square/go-jose: compact JWE format must have five parts","errorVerbose":"square/go-jose: compact JWE format must have five parts\nerror parsing provisioner encrypted key\ngithub.com/smallstep/certificates/ca.decryptProvisionerJWK\n\t/go/pkg/mod/github.com/smallstep/certificates@v0.23.2/ca/provisioner.go:158\ngithub.com/smallstep/certificates/ca.loadProvisionerJWKByKid\n\t/go/pkg/mod/github.com/smallstep/certificates@v0.23.2/ca/provisioner.go:179\ngithub.com/smallstep/certificates/ca.NewProvisioner\n\t/go/pkg/mod/github.com/smallstep/certificates@v0.23.2/ca/provisioner.go:54\ngithub.com/smallstep/step-issuer/provisioners.NewFromStepClusterIssuer\n\t/src/provisioners/step.go:61\ngithub.com/smallstep/step-issuer/controllers.(*StepClusterIssuerReconciler).Reconcile\n\t/src/controllers/stepclusterissuer_controller.go:91\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598","stacktrace":"github.com/smallstep/step-issuer/controllers.(*StepClusterIssuerReconciler).Reconcile\n\t/src/controllers/stepclusterissuer_controller.go:93\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}

The YAML config for the deployment is attached as "step-issuer.values.txt".
It seems like the StepClusterIssuer information is incomplete (or wrong), but the Step-Certificates instance is issuing certificates correctly.

[maraino]

From the error, it looks like the provisioner JWK provisioner with kid B5MjjDUqy64XitU1lEQ06WEt4UL2H1VZi-_UpYybB58 doesn't have a valid encryptedKey value. At least that's my guess seeing this error compact JWE format must have five parts. That encryptedKey is generally in the ca.json, although it can be stored in a database if step-ca is configured with it.

You can get the encryptedKey value using step ca provisioner list, and you can make sure it is properly formatted if you see a private key in JWK format when you type:

echo <encyrptedKey> | step crypto jwe decrypt

Using the password from the secret ecdsa-iss-step-certificates-provisioner-password.password.

PS: kid, if provided, is used by default instead of the name aksissuer.

[njunot]

Hello,

I am encountering the same issue. Thanks to your tip, I think I pinpointed the issue:

In my setup, the encrypted key is stored in the ca.json file in its corresponding JWK provisioner. Like in the documentation, it stored as a Base64 encoded string.

Once, the step-issuer pod is launched, it sends out a GET request to the step-CA pod for the encrypted key. That request is processed correctly by Step-CA:

method=GET name=ca path=/provisioners/3XfvieDi17HzTxEQB5LQmN9fpiygXIafWE4n2fhr1hE/encrypted-key

Meanwhile Step-Issuer logs the following error:

"msg":"failed to initialize provisioner","stepissuer":{"name":"calico-issuer","namespace":"calico-issuer"},"error":"error parsing provisioner encrypted key: go-jose/go-jose: compact JWE format must have five parts"...

So, I figure that Step-Issuer receives the key but cannot decrypt it.

After getting the encrypted key as it is stored in ca.json and feeding it to step crypto jwe decrypt, the tool yields the same error:

error parsing data: go-jose/go-jose: compact JWE format must have five parts

However, if that string is first decoded and then fed to step crypto jwe decrypt, the tool prompts for the password and afterwards the unencrypted key is returned.

This leads me to believe that Step-Issuer tries to decrypt the Base64 encoded string that it receives from Step-CA. Unfortunately, the decryption function doesn't find the five parts of the encrypted key it expects to receive. To work correctly, that Base64 string should first be decoded and then passed to the decryption function.

Kubernetes version: 1.32.3
Helm version: v3.17.3
StepCA Helm chart version: 1.28.3
CertManager Helm chart version: v1.17.1
StepIssuer Helm chart version: 1.9.8

Here is the complete error:

{"logger":"controllers.StepIssuer","msg":"failed to initialize provisioner","stepissuer":{"name":"step-issuer","namespace":"step-issuer"},"error":"error parsing provisioner encrypted key: go-jose/go-jose: compact JWE format must have five parts","errorVerbose":"go-jose/go-jose: compact JWE format must have five parts\nerror parsing provisioner encrypted key\ngithub.com/smallstep/certificates/ca.decryptProvisionerJWK\n\t/go/pkg/mod/github.com/smallstep/certificates@v0.28.2/ca/provisioner.go:160\ngithub.com/smallstep/certificates/ca.loadProvisionerJWKByKid\n\t/go/pkg/mod/github.com/smallstep/certificates@v0.28.2/ca/provisioner.go:181\ngithub.com/smallstep/certificates/ca.NewProvisioner\n\t/go/pkg/mod/github.com/smallstep/certificates@v0.28.2/ca/provisioner.go:56\ngithub.com/smallstep/step-issuer/provisioners.NewFromStepIssuer\n\t/src/provisioners/step.go:35\ngithub.com/smallstep/step-issuer/controllers.(*StepIssuerReconciler).Reconcile\n\t/src/controllers/stepissuer_controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.3/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.3/pkg/internal/controller/controller.go:328\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.3/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.3/pkg/internal/controller/controller.go:249\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700","stacktrace":"github.com/smallstep/step-issuer/controllers.(*StepIssuerReconciler).Reconcile\n\t/src/controllers/stepissuer_controller.go:92\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.3/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.3/pkg/internal/controller/controller.go:328\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.3/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.3/pkg/internal/controller/controller.go:249"}