Secret in imported namespace is wiped after each successful reconcile with manually created service account token

[nedenwalker]

What steps did you take:

kind create cluster --image kindest/node:v1.21.1
kubectl cluster-info --context kind-kind

kapp deploy -a sg -y \
  -f https://github.com/vmware-tanzu/carvel-secretgen-controller/releases/download/v0.5.0/release.yml

kubectl create namespace src-ns
kubectl apply -f - -o yaml << EOF
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: client
  namespace: src-ns

---
apiVersion: v1
kind: Secret
metadata:
  name: client-token
  namespace: src-ns
  annotations:
    kubernetes.io/service-account.name: client
type: kubernetes.io/service-account-token

---
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretExport
metadata:
  name: client-token
  namespace: src-ns
spec:
  toNamespace: dest-ns
EOF

kubectl create namespace dest-ns
kubectl apply -f - -o yaml << EOF
---
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretImport
metadata:
  name: client-token
  namespace: dest-ns
spec:
  fromNamespace: src-ns
EOF

What happened:
SecretImport flaps between ReconcileSucceeded and ReconcileFailed with Condition.Message Updating imported secret: Operation cannot be fulfilled on secrets "client-token": StorageError: invalid object, Code: 4, Key: /registry/secrets/dest-ns/client-token, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 69337dc5-9141-486a-aea4-a0fad920a4f9, UID in object meta:

The empty "UID in object meta" is certainly suspicious.

What did you expect:
The secret seems to be properly created in the destination namespace, but I expected the secret to stay created! See below for terminal output (certs and tokens partially redacted but confirmed they are the same in both src and dest namespace secrets).

Anything else you would like to add:
I don't know enough about ServiceAccounts and manually replacing the auto-generated ServiceAccountToken to understand what is going on here, maybe I am doing something I should not. Most documentation seems to focus on creating a ServiceAccount and just using the auto-generated token, but since we want to share the token into another namespace, it seemed we needed to create a service account token with an explicit name that we could then export/import.

Environment:

• Kind Cluster
• Secret Generator Controller version: v0.5.0
• Kubernetes version: Client v1.22.1; Server v1.21.1

(kind-kind:default) workspace $ kubectl get secretexport client-token -n src-ns -o yaml
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretExport
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"secretgen.carvel.dev/v1alpha1","kind":"SecretExport","metadata":{"annotations":{},"name":"client-token","namespace":"src-ns"},"spec":{"toNamespace":"dest-ns"}}
  creationTimestamp: "2021-10-17T15:08:58Z"
  generation: 1
  name: client-token
  namespace: src-ns
  resourceVersion: "76753"
  uid: ab510113-4e13-4689-97a1-9925bd742dda
spec:
  toNamespace: dest-ns
status:
  conditions:
  - status: "True"
    type: ReconcileSucceeded
  friendlyDescription: Reconcile succeeded
  observedGeneration: 1
  observedSecretResourceVersion: "76752"

(kind-kind:default) workspace $ kubectl get secret client-token -n src-ns -o yaml
apiVersion: v1
data:
  ca.crt: LS0tLS1C... ...LS0tCg==
  namespace: c3JjLW5z
  token: ZXlKaGJH... ...cFJHaTJB
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{"kubernetes.io/service-account.name":"client"},"name":"client-token","namespace":"src-ns"},"type":"kubernetes.io/service-account-token"}
    kubernetes.io/service-account.name: client
    kubernetes.io/service-account.uid: 8fd243b4-a3dd-4ec4-9a4f-131e3ebbe0c7
  creationTimestamp: "2021-10-17T15:08:58Z"
  name: client-token
  namespace: src-ns
  resourceVersion: "76752"
  uid: 6dd50965-30ac-467a-85f5-8a55a5ad70f4
type: kubernetes.io/service-account-token



(kind-kind:default) workspace $ kubectl get secretimport client-token -n dest-ns -o yaml
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretImport
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"secretgen.carvel.dev/v1alpha1","kind":"SecretImport","metadata":{"annotations":{},"name":"client-token","namespace":"dest-ns"},"spec":{"fromNamespace":"src-ns"}}
  creationTimestamp: "2021-10-17T15:08:58Z"
  generation: 1
  name: client-token
  namespace: dest-ns
  resourceVersion: "80211"
  uid: 92e61056-a386-453d-bb67-a9bf24c715cb
spec:
  fromNamespace: src-ns
status:
  conditions:
  - status: "True"
    type: ReconcileSucceeded
  friendlyDescription: Reconcile succeeded
  observedGeneration: 1

(kind-kind:default) workspace $ kubectl get secret client-token -n dest-ns -o yaml
apiVersion: v1
data:
  ca.crt: LS0tLS1C... ...LS0tCg==
  namespace: c3JjLW5z
  token: ZXlKaGJH... ...cFJHaTJB
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"secretgen.carvel.dev/v1alpha1","kind":"SecretImport","metadata":{"annotations":{},"name":"client-token","namespace":"dest-ns"},"spec":{"fromNamespace":"src-ns"}}
    kubernetes.io/service-account.name: client
    kubernetes.io/service-account.uid: 8fd243b4-a3dd-4ec4-9a4f-131e3ebbe0c7
  creationTimestamp: "2021-10-17T15:16:33Z"
  name: client-token
  namespace: dest-ns
  ownerReferences:
  - apiVersion: secretgen.carvel.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: SecretImport
    name: client-token
    uid: 92e61056-a386-453d-bb67-a9bf24c715cb
  resourceVersion: "90285"
  uid: fc933513-8571-44ff-a666-1f38ed4d48b4
type: kubernetes.io/service-account-token

(kind-kind:default) workspace $ kubectl get secretimport client-token -n dest-ns -o yaml
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretImport
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"secretgen.carvel.dev/v1alpha1","kind":"SecretImport","metadata":{"annotations":{},"name":"client-token","namespace":"dest-ns"},"spec":{"fromNamespace":"src-ns"}}
  creationTimestamp: "2021-10-17T15:08:58Z"
  generation: 1
  name: client-token
  namespace: dest-ns
  resourceVersion: "80291"
  uid: 92e61056-a386-453d-bb67-a9bf24c715cb
spec:
  fromNamespace: src-ns
status:
  conditions:
  - message: 'Updating imported secret: Operation cannot be fulfilled on secrets "client-token":
      StorageError: invalid object, Code: 4, Key: /registry/secrets/dest-ns/client-token,
      ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition:
      69337dc5-9141-486a-aea4-a0fad920a4f9, UID in object meta: '
    status: "True"
    type: ReconcileFailed
  friendlyDescription: 'Reconcile failed: Updating imported secret: Operation cannot
    be fulfilled on sec...'
  observedGeneration: 1

(kind-kind:default) workspace $ kubectl get secret client-token -n dest-ns -o yaml
Error from server (NotFound): secrets "client-token" not found

Additionally, the error from the secretgen-controller logs:
{"level":"error","ts":1634525504.5472927,"logger":"controller-runtime.manager.controller.sg-secimp","msg":"Reconciler error","name":"client-token","namespace":"dest-ns","error":"Updating imported secret: Operation cannot be fulfilled on secrets \"client-token\": StorageError: invalid object, Code: 4, Key: /registry/secrets/dest-ns/client-token, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 1b82f350-b28b-47e0-beff-f121a5ef4ac5, UID in object meta: ","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tgithub.com/go-logr/zapr@v0.2.0/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:267\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1\n\tsigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\tk8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\tk8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\tk8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:99"}

[nedenwalker]

Added commentary: we have moved away from trying to do the above for our use case, so we are not blocked on this. But the bug remains, so leaving this issue.