6.1.0 core nodes still remian after the cluster is upgraded to enterprise 6.2.0.

[Le9Wang]

Deployed an EMQX cluster with enterprise 6.1.0 through EMQX operator.
Try to upgrade the cluster to 6.2.0, the 6.1.0 core nodes still remian after the cluster is upgraded to enterprise 6.2.0.

cat emqx_6.yaml 
apiVersion: apps.emqx.io/v2
kind: EMQX
metadata:
   name: emqx-ee-6
   namespace: emqx-operator
spec:
   image: emqx/emqx-enterprise:6.2.0
   coreTemplate:
      spec:
         replicas: 2
   replicantTemplate:
      spec:
         replicas: 2
   config:
      data: |
         license {
         key = "evaluation"
         }

EMQX Operator

Log

2026-04-09T01:49:51Z ERROR Reconciler error {"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX", "EMQX": {"name":"emqx-ee-6","namespace":"emqx-operator"}, "namespace": "emqx-operator", "name": "emqx-ee-6", "reconcileID": "aa81a89d-9d54-457e-9f50-215696c60427", "error": "failed to forget DS site C928D1680F93A1EA: error accessing emqx-ee-6-core-75f6c6cdfd-0 API http://10.50.3.88:18083/api/v5/ds/sites/C928D1680F93A1EA/forget: request failed: Put "http://10.50.3.88:18083/api/v5/ds/sites/C928D1680F93A1EA/forget\": net/http: HTTP/1.x transport connection broken: malformed MIME header line: Content-Length: \u0000", "errorVerbose": "Put "http://10.50.3.88:18083/api/v5/ds/sites/C928D1680F93A1EA/forget\": net/http: HTTP/1.x transport connection broken: malformed MIME header line: Content-Length: \u0000\nrequest failed\ngithub.com/emqx/emqx-operator/internal/requester.(*Requester).Request\n\t/workspace/internal/requester/requester.go:111\ngithub.com/emqx/emqx-operator/internal/emqx/api.request\n\t/workspace/internal/emqx/api/api.go:60\ngithub.com/emqx/emqx-operator/internal/emqx/api.ForgetDSSite\n\t/workspace/internal/emqx/api/ds.go:170\ngithub.com/emqx/emqx-operator/internal/controller.(*dsCleanupSites).reconcile\n\t/workspace/internal/controller/ds_cleanup_sites.go:48\ngithub.com/emqx/emqx-operator/internal/controller.(*EMQXReconciler).Reconcile\n\t/workspace/internal/controller/emqx_controller.go:145\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\nerror accessing emqx-ee-6-core-75f6c6cdfd-0 API http://10.50.3.88:18083/api/v5/ds/sites/C928D1680F93A1EA/forget\ngithub.com/emqx/emqx-operator/internal/emqx/api.request\n\t/workspace/internal/emqx/api/api.go:62\ngithub.com/emqx/emqx-operator/internal/emqx/api.ForgetDSSite\n\t/workspace/internal/emqx/api/ds.go:170\ngithub.com/emqx/emqx-operator/internal/controller.(*dsCleanupSites).reconcile\n\t/workspace/internal/controller/ds_cleanup_sites.go:48\ngithub.com/emqx/emqx-operator/internal/controller.(*EMQXReconciler).Reconcile\n\t/workspace/internal/controller/emqx_controller.go:145\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\nfailed to forget DS site C928D1680F93A1EA\ngithub.com/emqx/emqx-operator/internal/controller.(*dsCleanupSites).reconcile\n\t/workspace/internal/controller/ds_cleanup_sites.go:50\ngithub.com/emqx/emqx-operator/internal/controller.(*EMQXReconciler).Reconcile\n\t/workspace/internal/controller/emqx_controller.go:145\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
2026-04-09T01:49:51Z DEBUG events reconcile failed at step dsCleanupSites, reason: failed to forget DS site C928D1680F93A1EA: error accessing emqx-ee-6-core-75f6c6cdfd-0 API http://10.50.3.88:18083/api/v5/ds/sites/C928D1680F93A1EA/forget: request failed: Put "http://10.50.3.88:18083/api/v5/ds/sites/C928D1680F93A1EA/forget": net/http: HTTP/1.x transport connection broken: malformed MIME header line: Content-Length: � {"type": "Warning", "object": {"kind":"EMQX","namespace":"emqx-operator","name":"emqx-ee-6","uid":"ec14d019-10a6-43d9-af3b-4fa92395981e","apiVersion":"apps.emqx.io/v2","resourceVersion":"265966629"}, "reason": "ReconcilerFailed"}
2026-04-09T01:49:52Z ERROR Reconciler error {"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX", "EMQX": {"name":"emqx-ee-6","namespace":"emqx-operator"}, "namespace": "emqx-operator", "name": "emqx-ee-6", "reconcileID": "5acb8689-33ee-4d44-9da2-2f91c88d97b9", "error": "failed to forget DS site 2AB3AE97BB3B79A2: error accessing emqx-ee-6-core-75f6c6cdfd-0 API http://10.50.3.88:18083/api/v5/ds/sites/2AB3AE97BB3B79A2/forget: request failed: Put "http://10.50.3.88:18083/api/v5/ds/sites/2AB3AE97BB3B79A2/forget": net/http: HTTP/1.x transport connection broken: malformed MIME header line: Content-Length: \u0000", "errorVerbose": "Put "http://10.50.3.88:18083/api/v5/ds/sites/2AB3AE97BB3B79A2/forget": net/http: HTTP/1.x transport connection broken: malformed MIME header line: Content-Length: \u0000\nrequest failed\ngithub.com/emqx/emqx-operator/internal/requester.(*Requester).Request\n\t/workspace/internal/requester/requester.go:111\ngithub.com/emqx/emqx-operator/internal/emqx/api.request\n\t/workspace/internal/emqx/api/api.go:60\ngithub.com/emqx/emqx-operator/internal/emqx/api.ForgetDSSite\n\t/workspace/internal/emqx/api/ds.go:170\ngithub.com/emqx/emqx-operator/internal/controller.(*dsCleanupSites).reconcile\n\t/workspace/internal/controller/ds_cleanup_sites.go:48\ngithub.com/emqx/emqx-operator/internal/controller.(*EMQXReconciler).Reconcile\n\t/workspace/internal/controller/emqx_controller.go:145\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\nerror accessing emqx-ee-6-core-75f6c6cdfd-0 API http://10.50.3.88:18083/api/v5/ds/sites/2AB3AE97BB3B79A2/forget\ngithub.com/emqx/emqx-operator/internal/emqx/api.request\n\t/workspace/internal/emqx/api/api.go:62\ngithub.com/emqx/emqx-operator/internal/emqx/api.ForgetDSSite\n\t/workspace/internal/emqx/api/ds.go:170\ngithub.com/emqx/emqx-operator/internal/controller.(*dsCleanupSites).reconcile\n\t/workspace/internal/controller/ds_cleanup_sites.go:48\ngithub.com/emqx/emqx-operator/internal/controller.(*EMQXReconciler).Reconcile\n\t/workspace/internal/controller/emqx_controller.go:145\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\nfailed to forget DS site 2AB3AE97BB3B79A2\ngithub.com/emqx/emqx-operator/internal/controller.(*dsCleanupSites).reconcile\n\t/workspace/internal/controller/ds_cleanup_sites.go:50\ngithub.com/emqx/emqx-operator/internal/controller.(*EMQXReconciler).Reconcile\n\t/workspace/internal/controller/emqx_controller.go:145\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...[]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
2026-04-09T01:49:52Z DEBUG events reconcile failed at step dsCleanupSites, reason: failed to forget DS site 2AB3AE97BB3B79A2: error accessing emqx-ee-6-core-75f6c6cdfd-0 API http://10.50.3.88:18083/api/v5/ds/sites/2AB3AE97BB3B79A2/forget: request failed: Put "http://10.50.3.88:18083/api/v5/ds/sites/2AB3AE97BB3B79A2/forget": net/http: HTTP/1.x transport connection broken: malformed MIME header line: Content-Length: � {"type": "Warning", "object": {"kind":"EMQX","namespace":"emqx-operator","name":"emqx-ee-6","uid":"ec14d019-10a6-43d9-af3b-4fa92395981e","apiVersion":"apps.emqx.io/v2","resourceVersion":"265966639"}, "reason": "ReconcilerFailed"}

[keynslug]

Thanks for the bug report @Le9Wang.

The issues with 6.1.0 -> 6.2.0 upgrade path, along with a couple that were uncovered during investigation were addressed in several PRs.

• chore: bump minirest to 1.4.12 emqx#17002
• fix(dsraft): use quorum machine upgrade strategy emqx#17055
• fix(dsraft): make drop DB propagate through cluster correctly emqx#17056
• test(upgrade): add upgrade workaround playbook #1190
• chore: restrict erts limits for nodetool invocations emqx#17144

Strictly speaking, it's a stretch to qualify this as a bug in Operator, it's more of an EMQX API design flaw that is going be addressed in 6.3.0. See #1190 for an update workaround. That said, I'll remove the label but keep it open for you to either close or follow up.