Skip to content

Raft-safe ClickHouseKeeperInstallation rescale #2041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
filimonov wants to merge 6 commits into
base: 0.27.3
Choose a base branch
from
+1,659 −31
Open

Raft-safe ClickHouseKeeperInstallation rescale #2041

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7748978
fix(types): ReconcileAttributes tag setters were silent no-ops on nil…
filimonov Jul 21, 2026
812321f
feat(chk): read-only Keeper membership observe client
filimonov Jul 21, 2026
23e7f1f
feat(chk): Lease leader election for the CHK controller manager
filimonov Jul 21, 2026
ee3a0e3
feat(chk): publish new Keeper members into the raft config one at a time
filimonov Jul 21, 2026
fb75da8
feat(chk): confirm committed Raft membership before advancing; safe s…
filimonov Jul 21, 2026
4578a13
test(chk): assert committed Raft membership in the keeper rescale e2e
filimonov Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions cmd/operator/app/thread_keeper.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package app

import (
"context"
"os"

"github.com/go-logr/logr"

Expand All @@ -20,6 +21,7 @@ import (
ctrlController "sigs.k8s.io/controller-runtime/pkg/controller"

api "github.com/altinity/clickhouse-operator/pkg/apis/clickhouse-keeper.altinity.com/v1"
deployment "github.com/altinity/clickhouse-operator/pkg/apis/deployment"
"github.com/altinity/clickhouse-operator/pkg/chop"
controller "github.com/altinity/clickhouse-operator/pkg/controller/chk"
)
Expand Down Expand Up @@ -67,6 +69,13 @@ func initKeeper(ctx context.Context) error {
// pkg/metrics/operator, and the CHK reconcile counters worth exposing are surfaced
// through that path, not through controller-runtime's manager-default exposition.
Metrics: metricsserver.Options{BindAddress: "0"},
// Serialize CHK reconciliation across operator replicas/restarts.
// The Lease lives in the operator's own namespace (empty value falls
// back to controller-runtime's in-cluster namespace detection).
LeaderElection: true,
LeaderElectionID: "clickhouse-keeper-operator.altinity.com",
LeaderElectionNamespace: os.Getenv(deployment.OPERATOR_POD_NAMESPACE),
LeaderElectionReleaseOnCancel: true,
})
if err != nil {
logger.Error(err, "init keeper - unable to ctrlRuntime.NewManager")
Expand Down
2 changes: 1 addition & 1 deletion config/chk/keeper_config.d/01-keeper-01-default-config.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
stricter list can override this value, but they must keep `ruok` if they
also use the default operator probes.
-->
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro</four_letter_word_white_list>
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro,rqld</four_letter_word_white_list>

@Slach Slach Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rqld is added to the default whitelist only — a user who overrides four_letter_word_white_list (the option documented a few lines above) will silently lose proactive leadership transfer on scale-down. That's acceptable since rqld is best-effort and the purge barrier is the real fence, but worth a note in the option's doc comment so users know to include rqld.

</keeper_server>
<listen_host>::</listen_host>
<listen_host>0.0.0.0</listen_host>
Expand Down
2 changes: 1 addition & 1 deletion deploy/builder/templates-config/chk/keeper_config.d/01-keeper-01-default-config.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
stricter list can override this value, but they must keep `ruok` if they
also use the default operator probes.
-->
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro</four_letter_word_white_list>
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro,rqld</four_letter_word_white_list>
</keeper_server>
<listen_host>::</listen_host>
<listen_host>0.0.0.0</listen_host>
Expand Down
17 changes: 17 additions & 0 deletions ...tes-install-bundle/clickhouse-operator-install-yaml-template-02-section-rbac-02-role.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,23 @@ rules:
- get
- list

#
# coordination.k8s.io resources
#

- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete

#
# apps.* resources
#
Expand Down
13 changes: 13 additions & 0 deletions .../clickhouse-operator/templates/generated/ClusterRole-clickhouse-operator-kube-system.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,19 @@ metadata:
labels: {{ include "altinity-clickhouse-operator.labels" . | nindent 4 }}
annotations: {{ include "altinity-clickhouse-operator.annotations" . | nindent 4 }}
rules:
# coordination.k8s.io resources (leader election)
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
#
# Core API group
#
Expand Down
13 changes: 13 additions & 0 deletions deploy/helm/clickhouse-operator/templates/generated/Role-clickhouse-operator.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,19 @@ metadata:
labels: {{ include "altinity-clickhouse-operator.labels" . | nindent 4 }}
annotations: {{ include "altinity-clickhouse-operator.annotations" . | nindent 4 }}
rules:
# coordination.k8s.io resources (leader election)
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
#
# Core API group
#
Expand Down
19 changes: 18 additions & 1 deletion deploy/operator/clickhouse-operator-install-ansible.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5496,6 +5496,23 @@ rules:
- get
- list

#
# coordination.k8s.io resources
#

- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete

#
# apps.* resources
#
Expand Down Expand Up @@ -6733,7 +6750,7 @@ data:
stricter list can override this value, but they must keep `ruok` if they
also use the default operator probes.
-->
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro</four_letter_word_white_list>
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro,rqld</four_letter_word_white_list>
</keeper_server>
<listen_host>::</listen_host>
<listen_host>0.0.0.0</listen_host>
Expand Down
34 changes: 31 additions & 3 deletions deploy/operator/clickhouse-operator-install-bundle-v1beta1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5365,7 +5365,6 @@ metadata:
namespace: kube-system
labels:
clickhouse.altinity.com/chop: 0.27.2

# Template Parameters:
#
# NAMESPACE=kube-system
Expand Down Expand Up @@ -5452,6 +5451,21 @@ rules:
- get
- list
#
# coordination.k8s.io resources
#
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
#
# apps.* resources
#
- apiGroups:
Expand Down Expand Up @@ -5618,7 +5632,6 @@ subjects:
- kind: ServiceAccount
name: clickhouse-operator
namespace: kube-system

# Template Parameters:
#
# NAMESPACE=kube-system
Expand Down Expand Up @@ -5705,6 +5718,21 @@ rules:
- get
- list
#
# coordination.k8s.io resources
#
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
#
# apps.* resources
#
- apiGroups:
Expand Down Expand Up @@ -6923,7 +6951,7 @@ data:
stricter list can override this value, but they must keep `ruok` if they
also use the default operator probes.
-->
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro</four_letter_word_white_list>
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro,rqld</four_letter_word_white_list>
</keeper_server>
<listen_host>::</listen_host>
<listen_host>0.0.0.0</listen_host>
Expand Down
36 changes: 35 additions & 1 deletion deploy/operator/clickhouse-operator-install-bundle.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5489,6 +5489,23 @@ rules:
- get
- list

#
# coordination.k8s.io resources
#

- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete

#
# apps.* resources
#
Expand Down Expand Up @@ -5755,6 +5772,23 @@ rules:
- get
- list

#
# coordination.k8s.io resources
#

- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete

#
# apps.* resources
#
Expand Down Expand Up @@ -6992,7 +7026,7 @@ data:
stricter list can override this value, but they must keep `ruok` if they
also use the default operator probes.
-->
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro</four_letter_word_white_list>
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro,rqld</four_letter_word_white_list>
</keeper_server>
<listen_host>::</listen_host>
<listen_host>0.0.0.0</listen_host>
Expand Down
18 changes: 16 additions & 2 deletions deploy/operator/clickhouse-operator-install-template-v1beta1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5365,7 +5365,6 @@ metadata:
namespace: ${OPERATOR_NAMESPACE}
labels:
clickhouse.altinity.com/chop: 0.27.2

# Template Parameters:
#
# NAMESPACE=${OPERATOR_NAMESPACE}
Expand Down Expand Up @@ -5452,6 +5451,21 @@ rules:
- get
- list
#
# coordination.k8s.io resources
#
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
#
# apps.* resources
#
- apiGroups:
Expand Down Expand Up @@ -6670,7 +6684,7 @@ data:
stricter list can override this value, but they must keep `ruok` if they
also use the default operator probes.
-->
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro</four_letter_word_white_list>
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro,rqld</four_letter_word_white_list>
</keeper_server>
<listen_host>::</listen_host>
<listen_host>0.0.0.0</listen_host>
Expand Down
19 changes: 18 additions & 1 deletion deploy/operator/clickhouse-operator-install-template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5489,6 +5489,23 @@ rules:
- get
- list

#
# coordination.k8s.io resources
#

- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete

#
# apps.* resources
#
Expand Down Expand Up @@ -6726,7 +6743,7 @@ data:
stricter list can override this value, but they must keep `ruok` if they
also use the default operator probes.
-->
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro</four_letter_word_white_list>
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro,rqld</four_letter_word_white_list>
</keeper_server>
<listen_host>::</listen_host>
<listen_host>0.0.0.0</listen_host>
Expand Down
19 changes: 18 additions & 1 deletion deploy/operator/clickhouse-operator-install-tf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5496,6 +5496,23 @@ rules:
- get
- list

#
# coordination.k8s.io resources
#

- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete

#
# apps.* resources
#
Expand Down Expand Up @@ -6733,7 +6750,7 @@ data:
stricter list can override this value, but they must keep `ruok` if they
also use the default operator probes.
-->
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro</four_letter_word_white_list>
<four_letter_word_white_list>conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro,rqld</four_letter_word_white_list>
</keeper_server>
<listen_host>::</listen_host>
<listen_host>0.0.0.0</listen_host>
Expand Down
Loading
Loading