How to add annotations to the pods in cluster

[MateoSegura]

Hello everyone, I am using vault side car injector feature to place the TLS certificates for my broker inside of each pod in a cluster deployment.

I've gotten this to work when I deploy a single instance of EMQX manually (meaning creating a k8s deployment by hand), but when I try to use the operator, I'm having trouble setting the annotations, here's what I have so far:

emqx-cluster.yaml

apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
  name: emqx-cluster
spec:
  image: emqx/emqx:5.3.0
  imagePullPolicy: IfNotPresent
  
  serviceAccountName: my-service-account
  
  coreTemplate:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'my-role'
        vault.hashicorp.com/agent-inject-secret-server.key: 'kv/secret'
        vault.hashicorp.com/agent-inject-template-server.key: |
          {{ `{{- with secret "kv/secret" -}}
          {{ .Data.private_key }}
          {{- end }}` }}

But when I look at my pods, the annotations are not there, and no side car containers are injected. When I run kubectl describe deployment I get this:

Name:             emqx-cluster-core-84f45d97f5-0
Namespace:        default
Priority:         0
Service Account:  mqtt-broker
Node:             ip-10-0-23-48.us-west-2.compute.internal/10.0.23.48
Start Time:       Fri, 03 Jan 2025 23:45:41 +0000
Labels:           apps.emqx.io/db-role=core
                  apps.emqx.io/instance=emqx-cluster
                  apps.emqx.io/managed-by=emqx-operator
                  apps.emqx.io/pod-template-hash=84f45d97f5
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=emqx-cluster-core-84f45d97f5-7ddbb6cb99
                  role=core
                  statefulset.kubernetes.io/pod-name=emqx-cluster-core-84f45d97f5-0
Annotations:      <none> # Notice how the annotations did not get added to the pod
Status:           Running
IP:               10.0.7.114
IPs:
  IP:           10.0.7.114
Controlled By:  StatefulSet/emqx-cluster-core-84f45d97f5
Containers:
  emqx:
    Container ID:   containerd://dff53c5c3b3f64de7acf33d6847cfbde0916cf296514e3bc863bfd8bda94ac53
    Image:          emqx/emqx:5.3.0
    Image ID:       docker.io/emqx/emqx@sha256:18298f5a7ce054cfab28e1d90081885615c3e8a7a1fcd92b4af596e46b649a65
    Ports:          1883/TCP, 8883/TCP, 18083/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 03 Jan 2025 23:51:42 +0000
      Finished:     Fri, 03 Jan 2025 23:51:45 +0000
    Ready:          False
    Restart Count:  6
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      1
      memory:   2Gi
    Liveness:   http-get http://:dashboard/status delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness:  http-get http://:dashboard/status delay=10s timeout=1s period=5s #success=1 #failure=12
    Environment:
      EMQX_DASHBOARD__LISTENERS__HTTP__BIND:  18083
      POD_NAME:                               emqx-cluster-core-84f45d97f5-0 (v1:metadata.name)
      EMQX_CLUSTER__DISCOVERY_STRATEGY:       dns
      EMQX_CLUSTER__DNS__RECORD_TYPE:         srv
      EMQX_CLUSTER__DNS__NAME:                emqx-cluster-headless.default.svc.cluster.local
      EMQX_HOST:                              $(POD_NAME).$(EMQX_CLUSTER__DNS__NAME)
      EMQX_NODE__DATA_DIR:                    data
      EMQX_NODE__ROLE:                        core
      EMQX_NODE__COOKIE:                      <set to the key 'node_cookie' in secret 'emqx-cluster-node-cookie'>  Optional: false
      EMQX_API_KEY__BOOTSTRAP_FILE:           "/opt/emqx/data/bootstrap_api_key"
    Mounts:
      /opt/emqx/data from emqx-cluster-core-data (rw)
      /opt/emqx/data/bootstrap_api_key from bootstrap-api-key (ro,path="bootstrap_api_key")
      /opt/emqx/etc/emqx.conf from bootstrap-config (ro,path="emqx.conf")
      /opt/emqx/log from emqx-cluster-core-log (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sl74s (ro)

[Rory-Z]

Hi, @MateoSegura What's the version of EMQX operator do you use ?

[MateoSegura]

Hello @Rory-Z, thanks for your response.

I am using emqx/emqx-operator-controller:2.2.26

[mwarnerdotme]

I'm also trying to get vault to inject authentication tokens into an emqx cluster deployment, but are running into similar issues where the annotations for the individual pods in that cluster are not being picked up by the emqx operator.

My understanding is that a cluster deployment will create instances of pods similar to how I'm creating an individual pod right now (by doing a custom deployment, and using Kubernetes annotations as specified by hashicorp here)

And when I look at my pod I can see the emqx container, as well as the vault injector side car container.

Is there a way to get this behavior to also work with the available CRDs offered by the emqx operator?

I'm using the same operator version as @MateoSegura

[Rory-Z]

Hi @MateoSegura @mwarnerdotme I'm sorry I didn't reappear your issue.

I'm install EMQX operator 2.2.26 in Kind, and create EMQX use follow YAML file:

apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
  name: emqx-cluster
spec:
  image: emqx/emqx:5.3.0
  imagePullPolicy: IfNotPresent
  coreTemplate:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'my-role'
        vault.hashicorp.com/agent-inject-secret-server.key: 'kv/secret'
    spec:
      replicas: 2

And check Pod's annotations, I got this:

kubectl get pods -l apps.emqx.io/instance=emqx-cluster -o yaml | yq '.items[0].metadata.annotations'

Looks good.

Could you please check EMQX's pod manager fields by kubectl get pods -l apps.emqx.io/instance=emqx-cluster --show-managed-fields -o yaml | yq '.items[0]' to make sure is it just EMQX operator changed Pod, or other manager changed it.

[mwarnerdotme]

Thanks for the help! Much appreciated.