Upgrade masters last when upgrading ES clusters #8871

naemono · 2025-10-22T18:38:05Z

Fixes #8429

What is changing?

This ensure that the master StatefulSets are always upgraded last when doing a version upgrade of Elasticsearch.

Validation

Manual testing
Unit test for behavior
e2e test validating behavior

Signed-off-by: Michael Montgomery <[email protected]>

prodsecmachine · 2025-10-22T18:38:22Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Licenses	0	0	0	0	0 issues
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Signed-off-by: Michael Montgomery <[email protected]>

naemono · 2025-10-29T01:58:29Z

buildkite test this -f p=kind,t=TestNonMasterFirstUpgradeComplexTopology -m s=9.1.2

Signed-off-by: Michael Montgomery <[email protected]>

updated. Signed-off-by: Michael Montgomery <[email protected]>

Signed-off-by: Michael Montgomery <[email protected]>

…/cloud-on-k8s into fix-sts-upgrade-issue-recreation

Signed-off-by: Michael Montgomery <[email protected]>

naemono · 2025-10-29T20:23:53Z

buildkite test this -f p=kind,t=TestHandleUpscaleAndSpecChanges_VersionUpgradeDataFirstFlow -m s=9.1.2

Signed-off-by: Michael Montgomery <[email protected]>

Copilot

Pull Request Overview

This PR implements a non-master-first upgrade strategy for Elasticsearch clusters. The key change ensures that during version upgrades, non-master nodes (data, ingest, coordinating nodes) are upgraded before master nodes, which helps maintain cluster stability during upgrades.

Adds logic to separate master and non-master StatefulSets during version upgrades
Implements upgrade order validation to ensure non-master nodes complete their upgrades first
Adds comprehensive unit and e2e tests to verify the upgrade flow

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
pkg/controller/elasticsearch/driver/upgrade.go	Adds check to identify new clusters vs upgrades by checking if status version is empty
pkg/controller/elasticsearch/driver/upscale.go	Implements non-master-first upgrade logic with resource separation and upgrade status checking
pkg/controller/elasticsearch/driver/upscale_test.go	Adds comprehensive unit test for version upgrade flow and minor formatting fixes
test/e2e/es/non_master_first_upgrade_test.go	Adds e2e test that validates non-master-first upgrade behavior with a watcher

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-29T23:29:59Z

test/e2e/es/non_master_first_upgrade_test.go

+				return
+			}
+
+			// Check if any master StatefulSet has it's version higher than any non-master StatefulSet


Corrected spelling of 'it's' to 'its' (possessive form, not contraction).

Suggested change

// Check if any master StatefulSet has it's version higher than any non-master StatefulSet

// Check if any master StatefulSet has its version higher than any non-master StatefulSet

test/e2e/es/non_master_first_upgrade_test.go

Signed-off-by: Michael Montgomery <[email protected]>

barkbay · 2025-11-03T09:47:34Z

pkg/controller/elasticsearch/driver/upscale.go

+		// If the StatefulSet is not at the target version, it is not upgraded
+		// so don't even bother looking at the state/status of the StatefulSet.


I think it is possible to enter a kind of "deadlock" when the master role is added to an existing nodeSet.
In that case:

The updated sts is detected here as a "not (yet) master sst"

But it is never going to be upgraded because it is considered as a masterResources in adjusted

Simple manifest to replicate the deadlock:

apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: elasticsearch-sample spec: version: 9.1.6 nodeSets: - name: ns1 config: node.roles: ["master","data"] node.store.allow_mmap: false count: 1 - name: ns2 config: node.roles: ["data", "ingest", "ml"] node.store.allow_mmap: false count: 3

Then upgrade to 9.2.0 with:

apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: elasticsearch-sample spec: version: 9.2.0 nodeSets: - name: ns1 config: node.roles: ["master","data"] node.store.allow_mmap: false count: 1 - name: ns2 config: node.roles: ["master", "data", "ingest", "ml"] node.store.allow_mmap: false count: 3

barkbay

I think we also have a problem when we increase the master replicas:

Initial manifest

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 9.1.6
  nodeSets:
  - name: ns1
    config:
      node.roles: ["master"]
      node.store.allow_mmap: false
    count: 1
  - name: ns2
    config:
      node.roles: ["data", "ingest", "ml"]
      node.store.allow_mmap: false
    count: 3

Updated manifest

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 9.2.0
  nodeSets:
  - name: ns1
    config:
      node.roles: ["master"]
      node.store.allow_mmap: false
    count: 3
  - name: ns2
    config:
      node.roles: ["data", "ingest", "ml"]
      node.store.allow_mmap: false
    count: 3

Operator is stuck in:

    inProgressOperations:
      downscale:
        lastUpdatedTime: "2025-11-03T10:04:05Z"
      upgrade:
        lastUpdatedTime: "2025-11-03T10:04:05Z"
      upscale:
        lastUpdatedTime: "2025-11-03T10:13:51Z"
        nodes:
        - message: Creating master node
          name: elasticsearch-sample-es-ns1-1
          status: EXPECTED
        - message: Limiting master nodes creation to one at a time
          name: elasticsearch-sample-es-ns1-2
          status: PENDING
    observedGeneration: 3
    phase: ApplyingChanges
    version: 9.1.6

barkbay · 2025-11-03T10:33:25Z

pkg/controller/elasticsearch/driver/upscale.go

+		// Check if this StatefulSet has pending updates
+		if statefulSet.Status.UpdatedReplicas != statefulSet.Status.Replicas {
+			return false, nil
+		}
+
+		// Check if there are any pods that need to be upgraded
+		pods, err := es_sset.GetActualPodsForStatefulSet(client, k8s.ExtractNamespacedName(&statefulSet))
+		if err != nil {
+			return false, err
+		}
+
+		for _, pod := range pods {
+			// Check if pod revision matches StatefulSet update revision
+			if statefulSet.Status.UpdateRevision != "" && sset.PodRevision(pod) != statefulSet.Status.UpdateRevision {
+				// This pod still needs to be upgraded
+				return false, nil
+			}
+		}
+	}


I think this whole logic assumes that the StatefulSet controller has updated the status, which is not necessarily true from my experiments. I hit several cases where the StatefulSet spec has been updated but not its status. A non master sts is then detected as reconciled while it is not the case.

barkbay · 2025-11-03T10:38:36Z

pkg/controller/elasticsearch/driver/upscale.go

+	}
+
+	// Check if all non-master StatefulSets have completed their upgrades before proceeding with master StatefulSets
+	if len(masterResources) > 0 {


I was wondering in which cases we are expecting this to be false?

Suggested change

if len(masterResources) > 0 {

Upgrade masters last when upgrading ES clusters

9db32d0

Signed-off-by: Michael Montgomery <[email protected]>

botelastic bot added the triage label Oct 22, 2025

naemono added 2 commits October 23, 2025 14:03

Fix lint issue

39b2702

Signed-off-by: Michael Montgomery <[email protected]>

Add e2e test for upgrade order.

50b3954

Signed-off-by: Michael Montgomery <[email protected]>

naemono added 5 commits October 28, 2025 21:00

unexport things in e2e tests

00555c2

Signed-off-by: Michael Montgomery <[email protected]>

Also look at the current/target version while determining whether sts is

88cb347

updated. Signed-off-by: Michael Montgomery <[email protected]>

Fix tests

790d3f1

Signed-off-by: Michael Montgomery <[email protected]>

Merge branch 'fix-sts-upgrade-issue-recreation' of github.com:naemono…

4b944d1

…/cloud-on-k8s into fix-sts-upgrade-issue-recreation

Fix the unit tests for master last upgrades

d9885ba

Signed-off-by: Michael Montgomery <[email protected]>

naemono added 5 commits October 29, 2025 18:03

fix linter

efa8643

Signed-off-by: Michael Montgomery <[email protected]>

move closer to use.

6914708

Signed-off-by: Michael Montgomery <[email protected]>

Ensure requeue

2dc664b

Signed-off-by: Michael Montgomery <[email protected]>

adjust comments

46c726c

Signed-off-by: Michael Montgomery <[email protected]>

Adjust logging in e2e test

fccf6c3

Signed-off-by: Michael Montgomery <[email protected]>

naemono changed the title ~~WIP: Upgrade masters last when upgrading ES clusters~~ Upgrade masters last when upgrading ES clusters Oct 29, 2025

naemono marked this pull request as ready for review October 29, 2025 23:21

naemono requested review from barkbay, Copilot, kvalliyurnatt, pebrc and rhr323 October 29, 2025 23:21

Copilot AI reviewed Oct 29, 2025

View reviewed changes

naemono added 2 commits October 30, 2025 15:21

Don't compare masters against other masters or themselves.

8feef24

Signed-off-by: Michael Montgomery <[email protected]>

Fix spelling

0f5a31a

Signed-off-by: Michael Montgomery <[email protected]>

barkbay added the >bug Something isn't working label Nov 3, 2025

botelastic bot removed the triage label Nov 3, 2025

barkbay added the v3.3.0 (next) label Nov 3, 2025

barkbay requested changes Nov 3, 2025

View reviewed changes

barkbay reviewed Nov 3, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Upgrade masters last when upgrading ES clusters #8871

Upgrade masters last when upgrading ES clusters #8871

naemono commented Oct 22, 2025 •

edited

Loading

Uh oh!

prodsecmachine commented Oct 22, 2025 •

edited

Loading

Uh oh!

naemono commented Oct 29, 2025

Uh oh!

naemono commented Oct 29, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Oct 29, 2025

Uh oh!

Uh oh!

barkbay Nov 3, 2025

Uh oh!

barkbay left a comment

Uh oh!

barkbay Nov 3, 2025

Uh oh!

barkbay Nov 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	// Check if any master StatefulSet has it's version higher than any non-master StatefulSet
	// Check if any master StatefulSet has its version higher than any non-master StatefulSet

		// If the StatefulSet is not at the target version, it is not upgraded
		// so don't even bother looking at the state/status of the StatefulSet.

Upgrade masters last when upgrading ES clusters #8871

Are you sure you want to change the base?

Upgrade masters last when upgrading ES clusters #8871

Conversation

naemono commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is changing?

Validation

Uh oh!

prodsecmachine commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

naemono commented Oct 29, 2025

Uh oh!

naemono commented Oct 29, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

barkbay Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

barkbay left a comment

Choose a reason for hiding this comment

Initial manifest

Updated manifest

Uh oh!

barkbay Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

barkbay Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

naemono commented Oct 22, 2025 •

edited

Loading

prodsecmachine commented Oct 22, 2025 •

edited

Loading