Controller crash with SecretTemplate that references other generated passwords

[petewall]

What steps did you take:
I'm trying to create a secret that references other generated secrets. The first secrets are created quickly, but the controller pod crashes when trying to resolve the fourth.

What happened:
The controller pod crashed

What did you expect:
The fourth secret to resolve, using data generated from the previous three

Anything else you would like to add:
Here are the inputs that I was using:

---
apiVersion: secretgen.k14s.io/v1alpha1
kind: SSHKey
metadata:
  name: worker-key
  namespace: concourse
spec: {}
---
apiVersion: secretgen.k14s.io/v1alpha1
kind: SSHKey
metadata:
  name: tsa-host-key
  namespace: concourse
spec: {}
---
apiVersion: secretgen.k14s.io/v1alpha1
kind: RSAKey
metadata:
  name: session-signing-key
  namespace: concourse
spec: {}
---
# Source: concourse/templates/web-secrets.yaml
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretTemplate
metadata:
  name: concourse-web
  namespace: concourse
spec:
  inputResources:
  - name: session-signing-key
    ref:
      apiVersion: v1
      kind: Secret
      name: session-signing-key
  - name: tsa-host-key
    ref:
      apiVersion: v1
      kind: Secret
      name: tsa-host-key
  - name: worker-key
    ref:
      apiVersion: v1
      kind: Secret
      name: worker-key
template:
  data:
    host-key: $(.tsa-host-key.data.privateKey)
    session-signing-key: $(.session-signing-key.data.privateKey)
    worker-key-pub: $(.worker-key.data.authorizedKey)
    local-users: "dGVzdDp0ZXN0"

The list of secrets:

$ kubectl get secret -n concourse
NAME                  TYPE                     DATA   AGE
session-signing-key   Opaque                   2      15m
worker-key            kubernetes.io/ssh-auth   2      15m
tsa-host-key          kubernetes.io/ssh-auth   2      15m
$ kubectl get secrettemplate -n concourse
NAME            DESCRIPTION   AGE
concourse-web   Reconciling   15m

And here are the controller pod logs:

pwall@Personal-MBP:~/src/petewall/cluster/deployments/concourse $ kubectl logs -n secretgen-controller secretgen-controller-667f6f9d67-zlt5m
{"level":"info","ts":1657380006.8805838,"logger":"sg.entrypoint","msg":"secretgen-controller","version":"0.10.3"}
{"level":"info","ts":1657380006.8806255,"logger":"sg.entrypoint","msg":"setting up manager"}
{"level":"info","ts":1657380007.2324445,"logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1657380007.2326853,"logger":"sg.entrypoint","msg":"setting up controllers"}
{"level":"info","ts":1657380007.2337837,"logger":"sg.entrypoint","msg":"starting manager"}
{"level":"info","ts":1657380007.2339766,"msg":"Starting metrics server","path":"/metrics"}
{"level":"info","ts":1657380007.2340772,"logger":"controller.sg-cert","msg":"Starting EventSource","source":"kind source: *v1alpha1.Certificate"}
{"level":"info","ts":1657380007.234097,"logger":"controller.sg-cert","msg":"Starting Controller"}
{"level":"info","ts":1657380007.2342255,"logger":"controller.sg-password","msg":"Starting EventSource","source":"kind source: *v1alpha1.Password"}
{"level":"info","ts":1657380007.2342393,"logger":"controller.sg-password","msg":"Starting Controller"}
{"level":"info","ts":1657380007.2342494,"logger":"controller.sg-secret","msg":"Starting EventSource","source":"kind source: *v1.Secret"}
{"level":"info","ts":1657380007.2342887,"logger":"controller.sg-secret","msg":"Starting EventSource","source":"kind source: *v1alpha1.SecretExport"}
{"level":"info","ts":1657380007.2342992,"logger":"controller.sg-secret","msg":"Starting EventSource","source":"kind source: *v1.Namespace"}
{"level":"info","ts":1657380007.2343037,"logger":"controller.sg-secret","msg":"Starting Controller"}
{"level":"info","ts":1657380007.2344172,"logger":"controller.sg-rsakey","msg":"Starting EventSource","source":"kind source: *v1alpha1.RSAKey"}
{"level":"info","ts":1657380007.2344322,"logger":"controller.sg-rsakey","msg":"Starting Controller"}
{"level":"info","ts":1657380007.2345197,"logger":"controller.sg-sshkey","msg":"Starting EventSource","source":"kind source: *v1alpha1.SSHKey"}
{"level":"info","ts":1657380007.2345417,"logger":"controller.sg-sshkey","msg":"Starting Controller"}
{"level":"info","ts":1657380007.234648,"logger":"controller.sg-template","msg":"Starting EventSource","source":"kind source: *v1.Secret"}
{"level":"info","ts":1657380007.2346628,"logger":"controller.sg-template","msg":"Starting EventSource","source":"kind source: *v1.Secret"}
{"level":"info","ts":1657380007.23467,"logger":"controller.sg-template","msg":"Starting EventSource","source":"kind source: *v1alpha1.SecretTemplate"}
{"level":"info","ts":1657380007.2346745,"logger":"controller.sg-template","msg":"Starting Controller"}
{"level":"info","ts":1657380007.2347345,"logger":"controller.sg-secexp","msg":"Starting EventSource","source":"kind source: *v1alpha1.SecretExport"}
{"level":"info","ts":1657380007.234757,"logger":"controller.sg-secexp","msg":"Starting EventSource","source":"kind source: *v1.Secret"}
{"level":"info","ts":1657380007.2347615,"logger":"controller.sg-secexp","msg":"Starting Controller"}
{"level":"info","ts":1657380007.2348795,"logger":"controller.sg-secimp","msg":"Starting EventSource","source":"kind source: *v1alpha1.SecretImport"}
{"level":"info","ts":1657380007.2348936,"logger":"controller.sg-secimp","msg":"Starting EventSource","source":"kind source: *v1.Secret"}
{"level":"info","ts":1657380007.234897,"logger":"controller.sg-secimp","msg":"Starting EventSource","source":"kind source: *v1alpha1.SecretExport"}
{"level":"info","ts":1657380007.2349007,"logger":"controller.sg-secimp","msg":"Starting EventSource","source":"kind source: *v1.Namespace"}
{"level":"info","ts":1657380007.2349072,"logger":"controller.sg-secimp","msg":"Starting Controller"}
{"level":"info","ts":1657380007.3345132,"logger":"controller.sg-password","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3346202,"logger":"controller.sg-cert","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.334624,"logger":"controller.sg-rsakey","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3347082,"logger":"controller.sg-sshkey","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3352313,"logger":"controller.sg-secret","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3354335,"logger":"sg.secret","msg":"Reconciling","request":"network/ddclient"}
{"level":"info","ts":1657380007.3355248,"logger":"controller.sg-template","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3355968,"logger":"controller.sg-secexp","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3356595,"logger":"sg.secret","msg":"Reconciling","request":"cert-manager/cert-manager-webhook-ca"}
{"level":"info","ts":1657380007.3357017,"logger":"controller.sg-secimp","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3356607,"logger":"sg.template","msg":"reconciling","request":"concourse/concourse-web"}
{"level":"info","ts":1657380007.335777,"logger":"sg.secexp","msg":"Reconciling","request":"cert-manager/letsencrypt-prod"}
{"level":"info","ts":1657380007.3357766,"logger":"sg.secret","msg":"Reconciling","request":"cert-manager/letsencrypt-prod"}
{"level":"info","ts":1657380007.3359263,"logger":"sg.secexp","msg":"Reconciling","request":"ghost/ghost-tls"}
{"level":"info","ts":1657380007.3359742,"logger":"sg.secexp","msg":"Reconciling","request":"concourse/session-signing-key"}
{"level":"info","ts":1657380007.3359776,"logger":"sg.secret","msg":"Reconciling","request":"ghost/ghost-tls"}
{"level":"info","ts":1657380007.3360217,"logger":"sg.secexp","msg":"Reconciling","request":"concourse/tsa-host-key"}
{"level":"info","ts":1657380007.3360746,"logger":"sg.secexp","msg":"Reconciling","request":"cert-manager/letsencrypt-staging"}
{"level":"info","ts":1657380007.336126,"logger":"sg.secret","msg":"Reconciling","request":"concourse/session-signing-key"}
{"level":"info","ts":1657380007.3361483,"logger":"sg.secexp","msg":"Reconciling","request":"concourse/worker-key"}
{"level":"info","ts":1657380007.3362415,"logger":"sg.secret","msg":"Reconciling","request":"concourse/tsa-host-key"}
{"level":"info","ts":1657380007.3363152,"logger":"sg.secret","msg":"Reconciling","request":"cert-manager/letsencrypt-staging"}
{"level":"info","ts":1657380007.336336,"logger":"sg.secexp","msg":"Reconciling","request":"network/ddclient"}
{"level":"info","ts":1657380007.336398,"logger":"sg.secret","msg":"Reconciling","request":"concourse/worker-key"}
{"level":"info","ts":1657380007.3364034,"logger":"sg.secexp","msg":"Reconciling","request":"cert-manager/cert-manager-webhook-ca"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x133637d]

goroutine 392 [running]:
github.com/vmware-tanzu/carvel-secretgen-controller/pkg/generator.evaluateTemplate(_, _)
        github.com/vmware-tanzu/carvel-secretgen-controller/pkg/generator/secret_template_reconciler.go:280 +0x3d
github.com/vmware-tanzu/carvel-secretgen-controller/pkg/generator.(*SecretTemplateReconciler).reconcile(0xc0005bea00, {0x18f4ec8, 0xc00057a570}, 0xc000497380)
        github.com/vmware-tanzu/carvel-secretgen-controller/pkg/generator/secret_template_reconciler.go:131 +0x8d
github.com/vmware-tanzu/carvel-secretgen-controller/pkg/generator.(*SecretTemplateReconciler).Reconcile(0xc0005bea00, {0x18f4ec8, 0xc00057a570}, {{{0xc0005a1d50?, 0x15d69c0?}, {0xc0005a1d30?, 0xc000046800?}}})
        github.com/vmware-tanzu/carvel-secretgen-controller/pkg/generator/secret_template_reconciler.go:121 +0x36c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc000433ae0, {0x18f4ec8, 0xc00057a4e0}, {{{0xc0005a1d50?, 0x15d69c0?}, {0xc0005a1d30?, 0xc0006bc5c0?}}})
        sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:114 +0x222
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000433ae0, {0x18f4e20, 0xc000046040}, {0x155fa40?, 0xc000490180?})
        sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:311 +0x2e9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000433ae0, {0x18f4e20, 0xc000046040})
        sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:266 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
        sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:223 +0x309

Environment:

• secretgen-controller version (execute kubectl get deployment -n secretgen-controller secretgen-controller -o yaml and the annotation is kbld.k14s.io/images):

    kbld.k14s.io/images: |
      - origins:
        - local:
            path: /home/runner/work/carvel-secretgen-controller/carvel-secretgen-controller
        - git:
            dirty: true
            remoteURL: https://github.com/vmware-tanzu/carvel-secretgen-controller
            sha: 7cf938231129673564646d851015d08630307efe
            tags:
            - v0.10.3
        url: ghcr.io/vmware-tanzu/carvel-secretgen-controller@sha256:00466d6beb98fdd8aed61642013ea0ba538bb496e84745c8c2e1871fdc54b1a9

• Kubernetes version (use kubectl version)

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.0-2+59bbb3530b6769", GitCommit:"59bbb3530b6769e4935a05ac0e13c9910c79253e", GitTreeState:"clean", BuildDate:"2022-05-13T06:41:13Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[petewall]

(I'll be changing local-users, so no worries about leaking the default test:test)

[petewall]

OK, figured out the crash. I mis-indented the SecretTemplate definition so template was parallel to spec and not inside it. Indenting that prevented the crash. Still probably shouldn't crash, so I'm going to leave this issue open.

Another maybe issue (let me know if you'd like me to open a new one), the key I needed to use to index the SSH and RSA secrets were not the same as is in the docs:
https://github.com/vmware-tanzu/carvel-secretgen-controller/blob/develop/docs/ssh_key.md
says I can use privateKey, but when it's generated, the secret uses ssh-privatekey.

[joe-kimmel-vmw]

Thanks @petewall ! I agree it sounds like we should update our docs, and likely there's a better path than crashing to handle the error case you're describing - appreciate your feedback!