Reconcile failure after deleting workload, several objects with invalid ownership metadata; annotation validation error

[a-abella]

Operator is installed from the grafana/k8s-monitoring v2 helm chart. I have two installed k8s-monitoring helm releases in the same namespace, therefore two operators in the namespace. I don't know if that could be causing some contention.

Experienced this error after deleting a statefulset managed by the operator,

Relevant part:

failed to install release: Unable to continue with install: ServiceAccount \"grafana-apm-monitoring-internaltempo-sampler-alloy\" in namespace \"grafana-alloy\" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key \"meta.helm.sh/release-name\": must be set to \"grafana-apm-monitoring-internaltempo-sampler\"; annotation validation error: missing key \"meta.helm.sh/release-namespace\": must be set to \"grafana-alloy\"

Same error regarding missing annotations occurred on ConfigMap, ClusterRole, ClusterRoleBinding, and Service objects.

Full trace:

{"level":"error","ts":"2025-06-07T02:54:33Z","logger":"helm.controller","msg":"Release failed","namespace":"grafana-alloy","name":"grafana-apm-monitoring-internaltempo-sampler","apiVersion":"collectors.grafana.com/v1alpha1","kind":"Alloy","release":"grafana-apm-monitoring-internaltempo-sampler","error":"failed to install release: Unable to continue with install: ServiceAccount \"grafana-apm-monitoring-internaltempo-sampler-alloy\" in namespace \"grafana-alloy\" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key \"meta.helm.sh/release-name\": must be set to \"grafana-apm-monitoring-internaltempo-sampler\"; annotation validation error: missing key \"meta.helm.sh/release-namespace\": must be set to \"grafana-alloy\"","stacktrace":"github.com/operator-framework/operator-sdk/internal/helm/controller.HelmOperatorReconciler.Reconcile\n\t/workspace/internal/helm/controller/reconcile.go:248\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-06-07T02:54:33Z","msg":"Reconciler error","controller":"alloy-controller","object":{"name":"grafana-apm-monitoring-internaltempo-sampler","namespace":"grafana-alloy"},"namespace":"grafana-alloy","name":"grafana-apm-monitoring-internaltempo-sampler","reconcileID":"d30125b1-760c-4135-bc47-03d41547a5af","error":"failed to install release: Unable to continue with install: ServiceAccount \"grafana-apm-monitoring-internaltempo-sampler-alloy\" in namespace \"grafana-alloy\" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key \"meta.helm.sh/release-name\": must be set to \"grafana-apm-monitoring-internaltempo-sampler\"; annotation validation error: missing key \"meta.helm.sh/release-namespace\": must be set to \"grafana-alloy\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224"}

Weird thing is that I deleted a different statefulset and that went fine.

I can see that the ServiceAccount associated with one StatefulSet correctly has its ownership annotations, the other doesn't:

Good, reconcile succeeded after deleting Statefulset:

Name:                grafana-apm-monitoring-oldtempo-sampler-alloy
Namespace:           grafana-alloy
Labels:              app.kubernetes.io/component=rbac
                     app.kubernetes.io/instance=grafana-apm-monitoring-oldtempo-sampler
                     app.kubernetes.io/managed-by=Helm
                     app.kubernetes.io/name=alloy
                     app.kubernetes.io/part-of=alloy
                     app.kubernetes.io/version=v1.8.3
                     helm.sdk.operatorframework.io/chart=alloy
                     helm.sh/chart=alloy-1.0.3
Annotations:         meta.helm.sh/release-name: grafana-apm-monitoring-oldtempo-sampler
                     meta.helm.sh/release-namespace: grafana-alloy

Bad, reconcile failed after deleting Statefulset:

Name:                grafana-apm-monitoring-internaltempo-sampler-alloy
Namespace:           grafana-alloy
Labels:              app.kubernetes.io/component=rbac
                     app.kubernetes.io/instance=grafana-apm-monitoring-internaltempo-sampler
                     app.kubernetes.io/managed-by=Helm
                     app.kubernetes.io/name=alloy
                     app.kubernetes.io/part-of=alloy
                     app.kubernetes.io/version=v1.8.3
                     helm.sdk.operatorframework.io/chart=alloy
                     helm.sh/chart=alloy-1.0.3
Annotations:         <none>

From my perspective as a user both these Alloys were created the same way, via the destinations object of the same k8s-monitoring helm values file.

[petewall]

This is interesting. I'm guessing there is an issue with multiple operators in the same namespace.
In your k8s-monitoring setup, you should be able to disable one of the operators and let the other deploy both k8s-monitoring instances' Alloys.

Or, to be more explicit, disable the operator in both, and deploy the alloy operator separately. This would prevent one of the k8s-monitoring being "dependent" on the other.

[petewall]

As for multiple instances, you can scope the alloy operator by namespace, which would help with installs to the different namespaces. But within the same namespace, I can't imagine a way to have multiple operators managing "their own" set of Alloy instances.