-
Notifications
You must be signed in to change notification settings - Fork 187
Description
Summary
This bug totally brokes pmm integration after upgrade chart, because pmm client container going into an infinite loop without start.
pmm-client:3.3.1 container fails to start into haproxy pod when deployed using pxc-db Helm chart version 1.18.0, due to the wrong key usage pmmserverkey into cluster-secret
that force the pxc operator 1.18.0 to choose pmm2 version instead of pmm3 that use key pmmservertoken as described into documentation
⸻
Affected Components
• pxc-db Helm chart: v1.18.0
• pxc-operator Helm chart v1.18.0
• pmm-client: v3.3.1 (default)
• Kubernetes: EKS v1.3x.x
⸻
Steps to Reproduce
- Deploy pxc operator 1.16.1 and deploy pxc-db 1.16.1 cluster with haproxy
- Upgrade pxc operator to 1.18.0 and upgrade pxc-db 1.18.0
- Observe logs from pmm-client sidecar in haproxy pods
Not tested but maybe bug triggered also without an upgrade but only setting pmmserverkey value
⸻
Expected Behavior
pmm-client should successfully going into a running state
⸻
Actual Behavior
The agent loops with:
pmm-agent is not running.
Config file /usr/local/percona/pmm2/config/pmm-agent.yaml is not writable: no such file or directory.
time="2025-10-07T15:31:43.912+00:00" level=info msg="'pmm-agent setup' exited with 1" component=entrypoint
⸻
Root Cause
pmm-client try to load a file that not exists because the pxc-operator inject wrong variable
PMM_AGENT_CONFIG_FILE: /usr/local/percona/pmm2/config/pmm-agent.yaml into haproxy statefulset, This behaviour is triggered from the parameter pmmserverkey that is the only available into cluster-secret
⸻
Proposed Fix
• Update pxc-db chart to inject also pmmservertoken value and make it as default
• Add validation logic for PMM client version alignment with secretname and image version
• Improve documentation for PMM compatibility
⸻
Impact
This issue causes pmm client container into PXC Cluster to not start anymore after un upgrade so basically you can broke your monitoring