KUBE-180: validate MongoDBSearch lifecycle recovery by anandsyncs · Pull Request #1383 · mongodb/mongodb-kubernetes

anandsyncs · 2026-07-16T01:58:43Z

Stack

Stack 3/4, based on #1382 (search-kube154-cleanup-self-healing) at accepted head 90e0eae8c5226ee8cb1259260aaf1d39fa675b3d (tree e3b69354fa29f1466222ec1f138fe2315bd26455). #1382 is stacked on #1362.

Head: search-kube154-lifecycle-e2e at 752df7eb5a0fa78512745f3696350bc4f18b6ba7 (tree 3ea52f6c8db276e45e4b15edaa68b7e6dd736644).

Commits

ebd8a55de8b9cf09c59128b7daca0324973ae2ac — KUBE-154: cover single-cluster Search lifecycle recovery
- docker/mongodb-kubernetes-tests/tests/common/search/connectivity.py
- docker/mongodb-kubernetes-tests/tests/common/search/search_resource_names.py
- docker/mongodb-kubernetes-tests/tests/search/search_community_basic.py
- docker/mongodb-kubernetes-tests/tests/search/search_enterprise_metrics_forwarder_replicaset.py
- docker/mongodb-kubernetes-tests/tests/search/search_external_metrics_forwarder_replicaset.py
- docker/mongodb-kubernetes-tests/tests/search/search_sharded_scale_shards_managed_lb.py
752df7eb5a0fa78512745f3696350bc4f18b6ba7 — KUBE-154: cover multi-cluster Search lifecycle recovery
- docker/mongodb-kubernetes-tests/kubetester/operator.py
- docker/mongodb-kubernetes-tests/tests/common/search/connectivity.py
- docker/mongodb-kubernetes-tests/tests/common/search/search_resource_names.py
- docker/mongodb-kubernetes-tests/tests/multicluster_search/q2_mc_rs_steady.py
- docker/mongodb-kubernetes-tests/tests/multicluster_search/simulated_mc_rs.py

Both commits retain the required Copilot co-author trailer. Their stable patch IDs remain dbeb4f0474c69857fbfc138719189fc5e2a750e2 and 2f2e1b4ae74ef602203725898c9616e3ebf44833, matching the reviewed logical commits exactly.

Scope

Community Search protected/user metadata coexistence, protected-label restoration, managed resource recreation, query continuity, and Search/PVC cleanup.
External and enterprise metrics-forwarder recreation/status/host convergence and deletion sequencing.
Sharded scale-down cleanup of stale mongot resources, generated TLS copies, and PVCs while preserving source TLS identity and Search behavior.
Simulated multi-cluster local mongot/Envoy recovery, operator restart barrier, ConfigMap recovery, query continuity, per-member cleanup, PVC cleanup, and protected input preservation.
Q2 real hub/member named single-member no-owner-reference shape, remote resource recreation, cross-member cleanup, PVC cleanup, and protected input UID preservation.
Existing canonical Search/operator helpers only; no new helper or e2e modules.

Non-scope

No Go or production changes.
No .evergreen.yml changes.
No tests/conftest.py changes.
No upgrade, pinned-GA, pre-GA, or upgrade-fixture wiring.
No changes to stack layers 1 or 2.

Local validation

Commit 1 independently: Black, isort, flake8, compileall, and full repository-backed NAMESPACE=collect-only uvx --offline --with-requirements ../../requirements.txt ty check tests/ passed; focused collection found 66 tests; existing Python tests passed 116 with scripts/release/tests/release_info_test.py::test_create_release_info_json deselected because it requires a Docker daemon; kubeobject passed 27.
Commit 2 independently: the same formatting, compile, full type, Python, and kubeobject checks passed; focused collection found 51 tests.
Cumulative head: the same checks passed across the complete diff; focused collection found 117 tests.
Fresh read-only MCK e2e review found no scope-blocking issues.

Evergreen

Final-base exhaustive patch 6a5a466cd981fd0007360290 is terminal. Its scope was inspected before activation and exactly matched the executable matrix: 57 requested targets across nine variants (all 56 executable Search targets plus unit_tests_golang) and 12 required dependencies, 69 tasks total. The result is 68 successes and one accepted deterministic failure: 56/57 requested targets and 12/12 dependencies succeeded.

Variant	Target scope	Count	Result
`e2e_mdb_kind_ubi_cloudqa_large`	All Search tasks from both Search task groups	34	all succeeded
`e2e_mdb_kind_ubi_cloudqa_search_block`	`e2e_search_block_multi_cluster`	1	succeeded
`e2e_mdb_openshift_search_ubi_cloudqa`	`e2e_search_sharded_openshift_external`	1	succeeded
`e2e_om80_kind_ubi_large`	Enterprise/external metrics-forwarder RS/sharded	4	all succeeded
`e2e_multi_cluster_kind`	Simulated MC RS/sharded	2	both succeeded
`e2e_multi_cluster_2_clusters`	Q2, Q3, MC connectivity RS/sharded, external MC RS/sharded	6	all succeeded
`e2e_operator_kind_ubi_cloudqa`	`e2e_operator_upgrade_search`	1	accepted pre-Layer-4 gap
`private_kind_code_snippets`	All `test_kind_search_*.sh` tasks	7	all succeeded
`unit_tests`	`unit_tests_golang`	1	succeeded

e2e_operator_upgrade_search failed executions 0 and 1 identically in TestOperatorUpgrade.test_database_running_after_upgrade after 300 seconds. In both executions, mdb-rs-2 remained at automation goal @-1 while mdb-rs-0 and mdb-rs-1 reached version 9. Search creation/query before upgrade, Search Running after upgrade, and mongot-version verification before and after upgrade all passed. This is the accepted pinned-GA/upgrade-fixture gap owned by Layer 4, not a Layer-3 lifecycle regression or infrastructure flake; no focused retry or Layer-3 change was made. Logs, test URLs, health snapshots, and both complete artifact bundles are preserved for Layer 4.

e2e_search_connectivity_tool_mc_sharded was included as an executable Search target and succeeded. Temporary patch-only activation was submitted only as uncommitted patch input and removed immediately; the branch remained clean.

Patch 6a5a382de34979000766218c used the superseded 56-request matrix and was canceled as non-evidence when the executable task set was corrected. Earlier patches 6a59e82a3ea0260007c2ee32, 6a589eb5586dd5000775cc0d, 6a583b98586dd5000773a9b9, and 6a587a6cf6311c00071550bf are also superseded and are not final validation evidence. Broad webhook patch 6a5a535d8f3f36000727c151 was canceled immediately and is non-evidence.

skip-ci

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

anandsyncs · 2026-07-20T08:39:07Z

+    ), f"expected at least one mongot data PVC for {names['sts']} before delete"
+
+
+def wait_for_search_artifacts_deleted(


Systemic tightening for all four cleanup tests: every absence check in this helper (and its callers) is by exact resource NAME. An orphan the operator leaves under an unexpected name — e.g. a stale TLS copy from a previous UID, or a legacy-named leftover — is invisible to every cleanup assertion in the stack. Adding one list-by-label-selector (mongodb.com/search-name=<name> [+ uid]) emptiness check per kind at the end of this helper closes that hole globally.

anandsyncs · 2026-07-20T08:39:11Z

+
+
+@mark.e2e_search_community_basic
+def test_search_statefulset_restores_removed_owner_label(namespace: str, mdbs: MongoDBSearch):


This waits only for the stripped mongodb.com/search-name label to return. A regression where the reapply restores that one label but drops search-uid/component or clobbers the custom user label would pass. After restoration, re-read once and assert the full protected set + CUSTOM_STATEFULSET_LABEL are all intact (reuse the tuple from the coexistence test above). Also 300s is generous for a next-reconcile reapply; 60-90s would catch latency regressions.

anandsyncs · 2026-07-20T08:39:16Z

-        if pvc.metadata.name.startswith(prefix)
-    ]
+@mark.e2e_search_community_basic
+def test_recreate_mongot_top_level_resources(


Recreated objects are validated by UID-change + readiness, but never re-checked for protected labels. A self-heal that recreates the STS without the owner labels (breaking future sweep/routing) passes. One extra assert on the recreated resources' labels closes it — same applies to the q2 and simulated-MC recreation tests.

anandsyncs · 2026-07-20T08:39:31Z

+        f"operator-managed TLS secret {operator_tls_secret} in {target_cluster_name}",
+        timeout=600,
+    )
+    wait_for_mongot_pvcs_deleted(namespace, sts_name, api_client=target_cluster.api_client, timeout=300)


Cross-talk gap: after deleting named_single, the test never re-asserts that mdbs's resources survived the sweep — the core 'label-scoped delete of CR A must not touch CR B' claim is only implicit in later tests. Re-read mdbs's STS/Envoy/TLS-secret here and assert present with unchanged UIDs.

anandsyncs · 2026-07-20T08:39:38Z

+        status = mdbs.get_metrics_forwarder_status()
+        return status is not None and status.get("phase") == Phase.Running.name
+
+    run_periodically(metrics_forwarder_running, timeout=120, interval=10)


run_periodically reads sleep_time=, not interval= — this kwarg is silently ignored and it polls every 2s. Harmless (still bounded) but copied from the pre-existing interval= sites in this file; worth fixing to sleep_time=10 (and adding msg=) here and in the siblings.

anandsyncs · 2026-07-20T08:39:43Z

Coverage-gap audit (from the stack review; ranked by customer risk). P0 — cheap, kind-testable, should be scheduled: (1) same-name delete→immediate-recreate of a Search (the UID adopt/reset path is a headline layer-1 behavior with zero e2e today); (2) disable-reconciliation annotation honored during delete-cleanup (fail-open if wrong). P1: upgrade e2e asserting legacy owner-ref removal + label backfill post-upgrade; CR-delete while operator is scaled to 0 → cleanup on restart; restart mid-deletion (sweep resume); MC member-cluster removal sweep (drop a clusters[] entry); sharded full-CR-delete cleanup (only scale-down is covered); single-cluster TLS-on lifecycle (TLS cleanup e2e is MC-only today). Member-unreachable fail-closed cleanup is EKS-only (kind's shared/multi-kind harness can't produce a clean partition) — cover that decision logic at unit level. Full detail in the stack reviewer guide.

anandsyncs force-pushed the search-kube154-lifecycle-e2e branch from 4fe0b46 to 6298544 Compare July 16, 2026 02:51

anandsyncs force-pushed the search-kube154-cleanup-self-healing branch from 9e05f7b to 7787462 Compare July 16, 2026 07:18

anandsyncs force-pushed the search-kube154-lifecycle-e2e branch from 6298544 to c2877b0 Compare July 16, 2026 08:51

anandsyncs force-pushed the search-kube154-cleanup-self-healing branch from 7787462 to 79421f8 Compare July 16, 2026 12:49

anandsyncs force-pushed the search-kube154-lifecycle-e2e branch from c2877b0 to a0f1469 Compare July 17, 2026 08:27

anandsyncs force-pushed the search-kube154-cleanup-self-healing branch from 79421f8 to 90e0eae Compare July 17, 2026 12:14

anandsyncs and others added 2 commits July 17, 2026 16:00

KUBE-154: cover single-cluster Search lifecycle recovery

ebd8a55

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

KUBE-154: cover multi-cluster Search lifecycle recovery

752df7e

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

anandsyncs force-pushed the search-kube154-lifecycle-e2e branch from a0f1469 to 752df7e Compare July 17, 2026 16:07

anandsyncs changed the title ~~KUBE-154: validate MongoDBSearch lifecycle recovery~~ KUBE-180: validate MongoDBSearch lifecycle recovery Jul 17, 2026

anandsyncs commented Jul 20, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

KUBE-180: validate MongoDBSearch lifecycle recovery#1383

KUBE-180: validate MongoDBSearch lifecycle recovery#1383
anandsyncs wants to merge 2 commits into
search-kube154-cleanup-self-healingfrom
search-kube154-lifecycle-e2e

anandsyncs commented Jul 16, 2026 •

edited

Loading

Uh oh!

anandsyncs Jul 20, 2026

Uh oh!

anandsyncs Jul 20, 2026

Uh oh!

anandsyncs Jul 20, 2026

Uh oh!

anandsyncs Jul 20, 2026

Uh oh!

anandsyncs Jul 20, 2026

Uh oh!

anandsyncs commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		), f"expected at least one mongot data PVC for {names['sts']} before delete"


		def wait_for_search_artifacts_deleted(



		@mark.e2e_search_community_basic
		def test_search_statefulset_restores_removed_owner_label(namespace: str, mdbs: MongoDBSearch):

Uh oh!

Conversation

anandsyncs commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stack

Commits

Scope

Non-scope

Local validation

Evergreen

Uh oh!

anandsyncs Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

anandsyncs Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

anandsyncs Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

anandsyncs Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

anandsyncs Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

anandsyncs commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

anandsyncs commented Jul 16, 2026 •

edited

Loading