Why update spec.config.data does not take effect?

[swechencheng]

Hi,
I am using emqx-operator to deploy the nodes:

apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
  name: emqx
  labels:
    apps.emqx.io/instance: emqx
    apps.emqx.io/managed-by: emqx-operator
spec:
  image: "emqx:5.3.0"
  imagePullPolicy: IfNotPresent
  revisionHistoryLimit: 3
  # imagePullSecrets:
  #   - name: fake-secrets
  updateStrategy:
    evacuationStrategy:
      connEvictRate: 1000
      sessEvictRate: 1000
      waitTakeover: 10
    initialDelaySeconds: 10
    type: Recreate
  config:
    data: |
      dashboard.listeners.http.bind = 18083
      log.console_handler.level = debug
      ...
      authentication = [
        ...
      ]
      authorization {
        ...
      }
  ...

After I did the first kubectl apply, everything works fine.

Then I am trying to follow this guide to change config via spec.config.data, e,.g., from log.console_handler.level = debug to log.console_handler.level = info
But nothing seems to happen after kubectl apply.

I have read configuration overrides rule and I don't use environment variable to override settings.
And I also checked kubectl exec emqx-core-574784d758-0 -- cat /opt/emqx/etc/emqx.conf and I see no change after kubectl apply with log.console_handler.level = info.

I have configured one core and one replicant. I have tried both emqx-operator 2.2.3 and 2.2.14.
But /opt/emqx/etc/emqx.conf never gets updated after I kubectl apply with spec.config.data changes.

Could anyone help me to understand why?
Is it possible if I insist to use spec.config.data for changing configuration, will emqx-operator take care of the rest after kubectl apply?

Thanks in advance!
BR/Chencheng

[alexa22013]

In my case on config.data update I also do some env changes to force restart, but the operator creates an entirely new cluster
operator logs:

{"level":"info","ts":"2024-02-08T14:08:43Z","msg":"got different pod template for EMQX core nodes, will create new statefulSet","controller":"emqx","controllerGroup":"apps.emqx.io","controllerKind":"EMQX","eMQX":{"name":"emqx","namespace":"emqx"},"namespace":"emqx","name":"emqx","reconcileID":"1621641b-059d-4fd9-a9a3-9e4c1f185889","statefulSet":{"name":"emqx-core-56db88d8b","namespace":"emqx"},"patch":"{\"spec\":{\"template\":{\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"emqx\"}],\"containers\":[{\"$setElementOrder/env\":[{\"name\":\"EMQX_DASHBOARD__LISTENERS__HTTP__BIND\"},{\"name\":\"POD_NAME\"},{\"name\":\"EMQX_CLUSTER__DISCOVERY_STRATEGY\"},{\"name\":\"EMQX_CLUSTER__DNS__RECORD_TYPE\"},{\"name\":\"EMQX_CLUSTER__DNS__NAME\"},{\"name\":\"EMQX_HOST\"},{\"name\":\"EMQX_NODE__DATA_DIR\"},{\"name\":\"EMQX_NODE__ROLE\"},{\"name\":\"EMQX_NODE__COOKIE\"},{\"name\":\"EMQX_API_KEY__BOOTSTRAP_FILE\"},{\"name\":\"RESTART_CLUSTER_DUMMY_ENV\"}],\"env\":[{\"name\":\"RESTART_CLUSTER_DUMMY_ENV\",\"value\":\"2\"},{\"$patch\":\"delete\",\"name\":\"EMQX_MQTT__MAX_INFLIGHT\"},{\"$patch\":\"delete\",\"name\":\"EMQX_MQTT__MAX_MQUEUE_LEN\"},{\"$patch\":\"delete\",\"name\":\"EMQX_MQTT__MQUEUE_STORE_QOS0\"},{\"$patch\":\"delete\",\"name\":\"EMQX_MQTT__PEER_CERT_AS_CLIENTID\"},{\"$patch\":\"delete\",\"name\":\"EMQX_MQTT__SESSION_EXPIRY_INTERVAL\"},{\"$patch\":\"delete\",\"name\":\"EMQX_MQTT__SHARED_SUBSCRIPTION_STRATEGY\"},{\"$patch\":\"delete\",\"name\":\"EMQX_PERSISTENT_SESSION_STORE__ENABLED\"}],\"name\":\"emqx\"}]}}}}"}
{"level":"info","ts":"2024-02-08T14:09:22Z","msg":"got different pod template for EMQX replicant nodes, will create new replicaSet","controller":"emqx","controllerGroup":"apps.emqx.io","controllerKind":"EMQX","eMQX":{"name":"emqx","namespace":"emqx"},"namespace":"emqx","name":"emqx","reconcileID":"18f7c10b-9f5e-4cca-9917-41eb94a958e4","replicaSet":{"name":"emqx-replicant-5857d7dc","namespace":"emqx"},"patch":"{\"spec\":{\"template\":{\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"emqx\"}],\"containers\":[{\"$setElementOrder/env\":[{\"name\":\"EMQX_DASHBOARD__LISTENERS__HTTP__BIND\"},{\"name\":\"EMQX_CLUSTER__DISCOVERY_STRATEGY\"},{\"name\":\"EMQX_CLUSTER__DNS__RECORD_TYPE\"},{\"name\":\"EMQX_CLUSTER__DNS__NAME\"},{\"name\":\"EMQX_HOST\"},{\"name\":\"EMQX_NODE__DATA_DIR\"},{\"name\":\"EMQX_NODE__ROLE\"},{\"name\":\"EMQX_NODE__COOKIE\"},{\"name\":\"EMQX_API_KEY__BOOTSTRAP_FILE\"},{\"name\":\"RESTART_CLUSTER_DUMMY_ENV\"}],\"env\":[{\"name\":\"RESTART_CLUSTER_DUMMY_ENV\",\"value\":\"2\"},{\"$patch\":\"delete\",\"name\":\"EMQX_MQTT__MAX_INFLIGHT\"},{\"$patch\":\"delete\",\"name\":\"EMQX_MQTT__MAX_MQUEUE_LEN\"},{\"$patch\":\"delete\",\"name\":\"EMQX_MQTT__MQUEUE_STORE_QOS0\"},{\"$patch\":\"delete\",\"name\":\"EMQX_MQTT__PEER_CERT_AS_CLIENTID\"},{\"$patch\":\"delete\",\"name\":\"EMQX_MQTT__SESSION_EXPIRY_INTERVAL\"},{\"$patch\":\"delete\",\"name\":\"EMQX_MQTT__SHARED_SUBSCRIPTION_STRATEGY\"},{\"$patch\":\"delete\",\"name\":\"EMQX_PERSISTENT_SESSION_STORE__ENABLED\"}],\"name\":\"emqx\"}]}}}}"}

How. to avoid this? operator: 2.2.3 emqx: 5.3.0

[swechencheng]

Even though I create an environment variable override EMQX_LOG__CONSOLE_HANDLER__LEVEL with another value, it does not take effect. :(

[Rory-Z]

Hi @alexa22013 Any action to update the Pod Template will result in creating a new Pod.

[Rory-Z]

Hi, @rtechencheng Sorry for delay, I just finished my vacation.
Could you please show the metadata.annotations.apps.emqx.io/last-emqx-configuration of you EMQX customer resource, and the configMap of $name-configs. If both them have log.console_handler.level = debug, please check is the EMQX pods has any error log.

[swechencheng]

Hi, @Rory-Z no worries.
I have upgraded to EMQX v5.4.1 and the problem seems to be gone now.