spec.pmm.pxcParams "--query-source=none" is silently overridden by hardcoded DB_ARGS=--query-source=perfschema

[Szuster6]

Description

The operator documentation states that spec.pmm.pxcParams can be used to pass custom pmm-admin add mysql
parameters, and community/forum guidance suggests --query-source=none disables Query Analytics (QAN)
collection. In practice, this parameter has no effect — the operator hardcodes
DB_ARGS=--query-source=perfschema as a container environment variable, and this value wins over
pxcParams (delivered via PMM_ADMIN_CUSTOM_PARAMS) due to argument ordering in the pmm-client
entrypoint command. As a result, QAN cannot be disabled through the CR, even though this is a
documented/expected capability.

Additional Context

• This makes it impossible to disable QAN collection for PXC-managed MySQL instances purely through the
  CR/Helm values, without resorting to a custom pmm-client image (overriding DB_ARGS at container
  entrypoint level) or a custom sidecar bypassing spec.pmm entirely.
• Motivation for wanting this: avoiding any additional load/queries against performance_schema on
  production database instances, independent of whether that load is actually significant.

Expected Results

Either:

• pxcParams should be able to override the operator's default --query-source value (e.g. by not
  hardcoding DB_ARGS, or by de-duplicating/ordering flags so user-supplied params win), or
• the CR should expose a dedicated field (e.g. spec.pmm.querySource: none|slowlog|perfschema) that
  is authoritative and not fought over by two separately-injected env vars.

Root Cause (source reference)

In pkg/pxc/app/statefulset/node.go, function PMM3PXCNodeEnvVars (two occurrences), DB_ARGS is a
hardcoded literal:

{
    Name:  "DB_ARGS",
    Value: "--query-source=perfschema",
},
{
    Name:  "PMM_ADMIN_CUSTOM_PARAMS",
    Value: PmmPxcParams,   // <- this is spec.pmm.pxcParams
},

(https://github.com/percona/percona-xtradb-cluster-operator/blob/main/pkg/pxc/app/statefulset/node.go)

In pkg/pxc/app/pmm.go, the pmm-client entrypoint command is built as:

pmmServerArgs := " $(PMM_ADMIN_CUSTOM_PARAMS) --skip-connection-check --metrics-mode=push"
...
if dbType == "mysql" {
    pmmServerArgs += " $(DB_ARGS)"
}

(https://github.com/percona/percona-xtradb-cluster-operator/blob/main/pkg/pxc/app/pmm.go)

$(PMM_ADMIN_CUSTOM_PARAMS) is placed before $(DB_ARGS) in the resulting pmm-admin add mysql
command. Since pmm-admin (kingpin-based CLI) takes the last occurrence of a repeated flag, the
operator's own --query-source=perfschema always wins over any user-supplied --query-source=none
passed via pxcParams.

Actual Results

The pmm-client container env shows both variables set, with the operator's hardcoded value present:

DB_ARGS=--query-source=perfschema
PMM_ADMIN_CUSTOM_PARAMS=--query-source=none

pmm-admin status still lists the QAN agent as active:

Agents:
    ...
    mysql_perfschema_agent   Running/Waiting   0

i.e. QAN collection continues, contrary to the --query-source=none request set via pxcParams.

Version

• Operator: percona-xtradb-cluster-operator 1.20.0 (helm chart pxc-operator 1.20.0)
• PXC image: percona/percona-xtradb-cluster:8.4.8-8.1 (MySQL 8.4)
• PMM Client image: percona/pmm-client:3.9.0
• PMM Server: 3.9.0
• Kubernetes: v1.36.2
• Deployment method: helm chart percona/pxc-db 1.20.0 (both via CR patch on an existing cluster AND via
  fresh helm install — same result in both cases, see reproduction)

Steps to reproduce

1. Deploy a PXC cluster with PMM enabled and pxcParams set to disable QAN:

   pmm:
     enabled: true
     image:
       tag: "3.9.0"
     serverHost: "<pmm-server-host>"
     pxcParams: "--query-source=none"

   (Reproduced both via kubectl patch pxc <name> --type merge -p '{"spec":{"pmm":{"pxcParams":"--query-source=none"}}}'
   on an existing cluster, and via a fresh helm install of the pxc-db chart with the above value
   present in values.yaml from the start — identical outcome in both cases.)
2. Wait for the pmm-client sidecar to be injected/reconciled.
3. Inspect the pmm-client container's environment variables:

   kubectl get pod <pxc-pod> -o jsonpath='{range .spec.containers[?(@.name=="pmm-client")].env[*]}{.name}={.value}{"\n"}{end}'

4. Check registered PMM agents:

   kubectl exec <pxc-pod> -c pmm-client -- pmm-admin status

Relevant logs

[pmm-agent@pxc-test4-pxc-0 ~]$ pmm-admin status
Agent ID : 5831032d-f35e-4270-b586-9f3567cc108e
Node ID  : 538b004e-943d-4197-b4c6-c142e072ad59
Node name: pxc-kopal-pxc-test4-pxc-0

PMM Server:
        URL    : <server>
        Version: 3.9.0

PMM Client:
        Connected        : true
        Time drift       : 1m8.963278402s
        Latency          : 546.563µs
        Connection uptime: 100
        pmm-admin version: 3.9.0
        pmm-agent version: 3.9.0
Agents:
        13fd8794-5ddf-4a95-ae3a-bc961f0263bf mysql_perfschema_agent         Running        0
        29f04850-74c4-4e48-91c2-56ed2f588983 mysqld_exporter                Running        30100
        8b4ce58b-c720-47eb-89e8-fa0b96f5c0d0 node_exporter                  Running        30101
        968c1f4c-df6a-4b7a-90d3-5516d4c5cb80 vmagent                        Running        30102

Code of Conduct

• I agree to follow Percona Community Code of Conduct