Add the ability to set labels and annotations per Flagd deployment

[Y0sh1dk]

Feature Request

Currently it is only possible to set labels and annotations globally via the operator helm values and not per Flagd deployment. It would be useful to have the ability to set labels and annotations via the Flagd CRD.

In my current setup, I am running multiple Flagd deployments and working around this limitation by using a Kyverno ClusterPolicy matching on labels to then patch in my desired annotations. A simplified version:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: mutate-flagd-resources
spec:
  rules:
    - name: set-flagd-client-side-resources
      match:
        resources:
          kinds:
            - Pod
          selector:
            matchLabels:
              app: client-side
              app.kubernetes.io/managed-by: open-feature-operator
      mutate:
        patchStrategicMerge:
          metadata:
            annotations:
              foo: bar

Proposed solution

1. Update the Flagd CRD to allow custom labels and annotations per individual Flagd instance.

apiVersion: core.openfeature.dev/v1beta1
kind: Flagd
metadata:
  name: flagd-sample
spec:
  replicas: 2
  podAnnotations: # Added to Deployment .spec.template.metadata.annotations
    prometheus.io/scrape: "true"
  podLabels: # Added to Deployment .spec.template.metadata.labels
    myLabel: "true"
  serviceType: ClusterIP
  serviceAccountName: default
  featureFlagSource: end-to-end
  ingress:
    enabled: true
    annotations:
      nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    hosts:
      - flagd-sample
    ingressClassName: nginx
    pathType: ImplementationSpecific

2. Copy Flagd labels and annotations onto resulting Deployment.

apiVersion: core.openfeature.dev/v1beta1
kind: Flagd
metadata:
  name: flagd-sample
  annotations: # Added to Deployment .spec.template.metadata.annotations
    prometheus.io/scrape: "true"
  labels: # Added to Deployment .spec.template.metadata.labels
    myLabel: "true"
spec:
  replicas: 2
  serviceType: ClusterIP
  serviceAccountName: default
  featureFlagSource: end-to-end
  ingress:
    enabled: true
    annotations:
      nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    hosts:
      - flagd-sample
    ingressClassName: nginx
    pathType: ImplementationSpecific

[Y0sh1dk]

Draft PR for the first proposed solution: #776