Skip to content

PMM 3 Client brokes deploy in pxc-db v1.18.0 after upgrade #662

@ynuyasha

Description

@ynuyasha

Summary

This bug totally brokes pmm integration after upgrade chart, because pmm client container going into an infinite loop without start.
pmm-client:3.3.1 container fails to start into haproxy pod when deployed using pxc-db Helm chart version 1.18.0, due to the wrong key usage pmmserverkey into cluster-secret
that force the pxc operator 1.18.0 to choose pmm2 version instead of pmm3 that use key pmmservertoken as described into documentation

Affected Components

• pxc-db Helm chart: v1.18.0
• pxc-operator Helm chart v1.18.0
• pmm-client: v3.3.1 (default)
• Kubernetes: EKS v1.3x.x

Steps to Reproduce

  1. Deploy pxc operator 1.16.1 and deploy pxc-db 1.16.1 cluster with haproxy
  2. Upgrade pxc operator to 1.18.0 and upgrade pxc-db 1.18.0
  3. Observe logs from pmm-client sidecar in haproxy pods

Not tested but maybe bug triggered also without an upgrade but only setting pmmserverkey value

Expected Behavior

pmm-client should successfully going into a running state

Actual Behavior

The agent loops with:
pmm-agent is not running.
Config file /usr/local/percona/pmm2/config/pmm-agent.yaml is not writable: no such file or directory.
time="2025-10-07T15:31:43.912+00:00" level=info msg="'pmm-agent setup' exited with 1" component=entrypoint

Root Cause

pmm-client try to load a file that not exists because the pxc-operator inject wrong variable
PMM_AGENT_CONFIG_FILE: /usr/local/percona/pmm2/config/pmm-agent.yaml into haproxy statefulset, This behaviour is triggered from the parameter pmmserverkey that is the only available into cluster-secret

Proposed Fix

• Update pxc-db chart to inject also pmmservertoken value and make it as default
• Add validation logic for PMM client version alignment with secretname and image version
• Improve documentation for PMM compatibility

Impact

This issue causes pmm client container into PXC Cluster to not start anymore after un upgrade so basically you can broke your monitoring

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions