Reconcile remote datacenters independently #2535

zimnx · 2025-03-05T16:46:05Z

Refactors ScyllaDBCluster controller to reconcile each datacenter independenly.
Errors or connection issues of down datacenter no longer affects reconcilation of other datacenter.

Requires:

Fixes #2494

scylla-operator-bot · 2025-03-05T16:46:08Z

@zimnx: GitHub didn't allow me to request PR reviews from the following users: zimnx.

Note that only scylladb members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

TODO

/cc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

zimnx · 2025-04-22T12:53:49Z

clean rebase to pick up dependent PRs, ready to review

mflendrich

My overall perspective - not reflected in the attached comments - is that the MakeRemoteXXX functions encapsulate significant logic in a way that (1) is not clear from the function name/signature, (2) vastly differs between types.

Attaching some comments, but in subsequent rounds I would like to get the function signatures (including mostly nonexistent docstrings) more in check with what actually the logic is.

pkg/controller/scylladbcluster/resource.go

pkg/controller/scylladbcluster/conditions.go

pkg/controller/scylladbcluster/sync.go

pkg/controller/scylladbcluster/conditions.go

rzetelskik

one minor comment and a nit
lgtm, but I second most of Michal's comments
/assign mflendrich

rzetelskik · 2025-04-29T15:31:23Z

pkg/controller/scylladbcluster/resource.go

@@ -6,6 +6,9 @@ import (
 	"sort"
 	"strings"

+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/apimachinery/pkg/util/errors"
+


blank line sneaked in

rzetelskik · 2025-04-29T16:08:58Z

test/e2e/set/scylladbcluster/multidatacenter/scylladbcluster_fault_tolerance.go

+		}
+
+		metaCluster := f.Cluster(0)
+		sc, err = f.ScyllaClient().ScyllaV1alpha1().ScyllaDBClusters(f.Namespace()).Create(ctx, sc, metav1.CreateOptions{})


nit: for clarity, shouldn't this use metaCluster instead of f directly as well?

indeed, done

pkg/controller/scylladbcluster/resource.go

pkg/controller/scylladbcluster/resource_test.go

pkg/controller/scylladbcluster/sync.go

pkg/internalapi/conditions.go

test/e2e/utils/verification/scylladbcluster/verify.go

Refactors ScyllaDBCluster controller to reconcile each datacenter independenly. Errors or connection issues of down datacenter no longer affects reconcilation of other datacenter.

mflendrich

The post-merge cleanup refactor promise, as decided in #2535 (comment), addresses my code health concern.

Two comments left, otherwise lgtm.

pkg/internalapi/conditions.go

pkg/controller/scylladbcluster/conditions.go

pkg/controller/scylladbcluster/resource.go

test/e2e/utils/verification/scylladbcluster/verify.go

pkg/controller/scylladbcluster/sync.go

…center Existing Controller Conditions were changed to represent status of reconcilation of particular Datacenter. On top of these, aggregated per Datacenter Conditions were added.

Test verifies if Operator is able to reconcile healthy datacenters when any is down, and whether it fully reconciles the cluster when down DC is restored.

mflendrich

/lgtm

Thanks!

scylla-operator-bot · 2025-05-06T11:57:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mflendrich, zimnx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mflendrich,zimnx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

zimnx force-pushed the independent-datacenters branch from b3235c0 to 433e060 Compare March 5, 2025 16:46

scylla-operator-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 5, 2025

zimnx force-pushed the independent-datacenters branch 2 times, most recently from 057822f to 61cac51 Compare March 10, 2025 16:21

scylla-operator-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 17, 2025

zimnx force-pushed the independent-datacenters branch from 61cac51 to 110646f Compare March 18, 2025 10:53

scylla-operator-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 18, 2025

zimnx force-pushed the independent-datacenters branch from 110646f to f8a89a1 Compare March 28, 2025 12:47

zimnx force-pushed the independent-datacenters branch 2 times, most recently from d4b239d to 978ed02 Compare April 9, 2025 13:43

zimnx changed the title ~~[WIP] Independently reconciled datacenters~~ Reconcile remote datacenters independently Apr 9, 2025

scylla-operator-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 9, 2025

zimnx added kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Apr 9, 2025

scylla-operator-bot bot removed do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Apr 9, 2025

zimnx changed the title ~~Reconcile remote datacenters independently~~ [WIP] Reconcile remote datacenters independently Apr 9, 2025

scylla-operator-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 9, 2025

zimnx force-pushed the independent-datacenters branch 5 times, most recently from 9519be5 to edf2b32 Compare April 15, 2025 09:22

scylla-operator-bot bot requested a review from rzetelskik April 17, 2025 11:34

zimnx force-pushed the independent-datacenters branch from 15fbeb0 to 606e28a Compare April 22, 2025 12:53

mflendrich reviewed Apr 22, 2025

View reviewed changes

mflendrich reviewed Apr 23, 2025

View reviewed changes

pkg/controller/scylladbcluster/sync.go Outdated Show resolved Hide resolved

pkg/controller/scylladbcluster/conditions.go Outdated Show resolved Hide resolved

scylla-operator-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 23, 2025

zimnx force-pushed the independent-datacenters branch from 606e28a to 645b2c5 Compare April 24, 2025 19:51

scylla-operator-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 24, 2025

zimnx requested a review from mflendrich April 24, 2025 20:02

zimnx force-pushed the independent-datacenters branch from 645b2c5 to 2a9c192 Compare April 25, 2025 08:32

rzetelskik reviewed Apr 29, 2025

View reviewed changes

scylla-operator-bot bot assigned mflendrich Apr 29, 2025

mflendrich reviewed Apr 29, 2025

View reviewed changes

zimnx force-pushed the independent-datacenters branch from 2a9c192 to b49b98c Compare May 2, 2025 08:48

Reconcile remote datacenters independently

bc9488f

Refactors ScyllaDBCluster controller to reconcile each datacenter independenly. Errors or connection issues of down datacenter no longer affects reconcilation of other datacenter.

zimnx force-pushed the independent-datacenters branch 2 times, most recently from 2e55354 to 5cdaf62 Compare May 5, 2025 10:20

zimnx mentioned this pull request May 5, 2025

Refactor ScyllaDBCluster controller resources specs #2622

Open

1 task

zimnx requested a review from mflendrich May 5, 2025 14:08

mflendrich reviewed May 5, 2025

View reviewed changes

zimnx force-pushed the independent-datacenters branch from 5cdaf62 to 93e4f72 Compare May 6, 2025 11:08

zimnx requested a review from mflendrich May 6, 2025 11:12

zimnx added 2 commits May 6, 2025 13:28

Add aggregated Conditions representing status of each reconciled Data…

d8a93a8

…center Existing Controller Conditions were changed to represent status of reconcilation of particular Datacenter. On top of these, aggregated per Datacenter Conditions were added.

Add fault tolerance E2E to Multidatacenter suite of ScyllaDBCluster

fd2b043

Test verifies if Operator is able to reconcile healthy datacenters when any is down, and whether it fully reconciles the cluster when down DC is restored.

zimnx force-pushed the independent-datacenters branch from 93e4f72 to fd2b043 Compare May 6, 2025 11:28

mflendrich approved these changes May 6, 2025

View reviewed changes

scylla-operator-bot bot added the lgtm Indicates that a PR is ready to be merged. label May 6, 2025

scylla-operator-bot bot merged commit d37be76 into scylladb:master May 6, 2025
13 checks passed

zimnx deleted the independent-datacenters branch May 6, 2025 12:44

Reconcile remote datacenters independently #2535

Reconcile remote datacenters independently #2535

Uh oh!

Conversation

zimnx commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scylla-operator-bot bot commented Mar 5, 2025

Uh oh!

zimnx commented Apr 22, 2025

Uh oh!

mflendrich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rzetelskik left a comment

Choose a reason for hiding this comment

Uh oh!

rzetelskik Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

zimnx Apr 30, 2025

Choose a reason for hiding this comment

Uh oh!

rzetelskik Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

zimnx Apr 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mflendrich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mflendrich left a comment

Choose a reason for hiding this comment

Uh oh!

scylla-operator-bot bot commented May 6, 2025

Uh oh!

Uh oh!

Uh oh!

zimnx commented Mar 5, 2025 •

edited

Loading