PMM-15151: make PG and ClickHouse operators watch all namespaces#868
Open
theTibi wants to merge 5 commits into
Open
PMM-15151: make PG and ClickHouse operators watch all namespaces#868theTibi wants to merge 5 commits into
theTibi wants to merge 5 commits into
Conversation
The pmm-ha-dependencies chart installed the PostgreSQL (Percona PGO) and ClickHouse (Altinity) operators scoped to their own namespace, so they did not reconcile PerconaPGCluster / ClickHouseInstallation CRs created in other namespaces. Installing pmm-ha into a second namespace (e.g. a DR restore target) left PostgreSQL and ClickHouse with no pods, while only VictoriaMetrics came up — its operator is already cluster-scoped. - pg-operator: set watchAllNamespaces=true. This renders WATCH_NAMESPACE="" (all namespaces) and switches the operator RBAC from a namespaced Role/RoleBinding to ClusterRole/ClusterRoleBinding. - altinity-clickhouse-operator: RBAC is already cluster-wide, but the operator only watched its own namespace; set WATCH_NAMESPACES=.* so it watches every namespace. (An empty watch list, documented as "all", did not take effect in 0.25.4.) Lets a DR/secondary namespace be added without manual operator edits. Validated by upgrading pmm-operators with these values: the PG and ClickHouse operators then reconciled CRs in a second namespace, with no disruption to the existing namespace. Signed-off-by: theTibi <tkorocz@gmail.com>
theTibi
requested review from
eleo007,
hors,
jvpasinatto,
nmarukovich and
valmiranogueira
as code owners
June 12, 2026 15:05
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the pmm-ha-dependencies Helm chart defaults so the PostgreSQL (Percona PGO) and ClickHouse (Altinity) operators reconcile custom resources across all namespaces, enabling multi-namespace PMM HA deployments (e.g., DR restore targets) to function correctly.
Changes:
- Configure
altinity-clickhouse-operatorto watch all namespaces by settingWATCH_NAMESPACES=.*. - Configure
pg-operatorfor cluster-wide operation by settingwatchAllNamespaces: true(which also switches RBAC to cluster-scoped in the subchart). - Add inline documentation in values to explain the rationale and behavioral details.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…elease name
The pmm.haPeers helper built the HA memberlist peer FQDNs from .Release.Name
(<release>-<ordinal>.monitoring-service...), but the PMM pods are named after the
StatefulSet (pmm.fullname): <release>-pmm-ha-<ordinal>. These only coincide when the
release is named "pmm-ha". With any other release name (e.g. a DR install named
"pmm-dr"), every peer name resolved to a non-existent host, the memberlist could not
form, and pmm-managed panicked on startup ("cannot start high availability service:
failed to join memberlist cluster ... no such host") — so PMM-HA never came up.
Build the peers from include "pmm.fullname" so they match the actual pod hostnames for
any release name. No change for a release named "pmm-ha" (fullname == release name).
Part of multi-namespace support: lets pmm-ha be installed under any release name (e.g. a
secondary/DR namespace) with a working HA cluster.
Signed-off-by: theTibi <tkorocz@gmail.com>
theTibi
requested review from
4nte and
JiriCtvrtka
and removed request for
a team
June 18, 2026 11:10
…, not the release name The HAProxy init script (haproxy-init-configmap.yaml) probes each PMM instance at <release>-<ordinal>.monitoring-service..., but the PMM pods are named after the StatefulSet (pmm.fullname): <release>-pmm-ha-<ordinal>. As with pmm.haPeers, these only coincide when the release is named "pmm-ha". With any other release name the probe targets non-existent hosts (HTTPS status 000), wait-for-pmm-ready never passes, and HAProxy never starts — even though PMM itself is healthy. Build pmm_host from include "pmm.fullname" so it matches the actual pod hostnames for any release name. Companion to the pmm.haPeers fix; completes multi-namespace support for PMM-HA. Signed-off-by: theTibi <tkorocz@gmail.com>
theTibi
requested review from
egegunes,
gkech,
mayankshah1607,
oksana-grishchenko and
pooknull
as code owners
July 3, 2026 16:10
Comment on lines
+98
to
+101
| {{- /* Peers must use the StatefulSet name (pmm.fullname), not Release.Name: the pods are | ||
| <fullname>-<ordinal> (e.g. pmm-dr-pmm-ha-0). <release>-<ordinal> only coincides with the | ||
| pod name when the release is named "pmm-ha"; any other release name yields peers that | ||
| don't resolve and the HA memberlist panics on startup. */}} |
Comment on lines
+19
to
+29
| # The operator's RBAC is already cluster-wide (rbac.namespaceScoped defaults to false), | ||
| # but it only watches its own namespace unless told otherwise — so ClickHouseInstallations | ||
| # in other namespaces (e.g. a DR restore target) are never reconciled. Make it watch all | ||
| # namespaces. NOTE: an empty watch list ("all namespaces" per the docs) did NOT take | ||
| # effect in 0.25.4 during testing (the operator stayed on its own namespace), so use a | ||
| # regexp that matches every namespace. Validate after deploy; alternatively set this to a | ||
| # comma-separated list/regexp of specific DR target namespaces. [PMM-15151] | ||
| operator: | ||
| env: | ||
| - name: WATCH_NAMESPACES | ||
| value: ".*" |
Comment on lines
+36
to
+41
| # Watch ALL namespaces so PostgreSQL clusters created in other namespaces — e.g. a | ||
| # cross-namespace DR restore target — are reconciled. This also switches the operator | ||
| # RBAC from a namespaced Role/RoleBinding to a ClusterRole/ClusterRoleBinding. Matches | ||
| # the VictoriaMetrics operator, which is already cluster-scoped. (Subchart default is | ||
| # false = the operator watches only its own namespace.) [PMM-15151] | ||
| watchAllNamespaces: true |
Comment on lines
49
to
51
| for i in $(seq 0 $(({{ .Values.replicas }} - 1))); do | ||
| pmm_host="{{ .Release.Name }}-$i.{{ .Values.service.name | default "monitoring-service" }}.{{ .Release.Namespace }}.svc.cluster.local" | ||
| pmm_host="{{ include "pmm.fullname" . }}-$i.{{ .Values.service.name | default "monitoring-service" }}.{{ .Release.Namespace }}.svc.cluster.local" | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The pmm-ha-dependencies chart installed the PostgreSQL (Percona PGO) and ClickHouse (Altinity) operators scoped to their own namespace, so they did not reconcile PerconaPGCluster / ClickHouseInstallation CRs created in other namespaces. Installing pmm-ha into a second namespace (e.g. a DR restore target) left PostgreSQL and ClickHouse with no pods, while only VictoriaMetrics came up — its operator is already cluster-scoped.