e2e scripts: make setup idempotent and fix test runner credentials by delthas · Pull Request #2363 · scality/Zenko

delthas · 2026-03-25T17:50:39Z

Summary

Make kubectl create calls idempotent with --dry-run=client | kubectl apply
Fix re-run failures for helm repos, root-ca, prometheus and kafka CRDs
Reuse existing RSA keypair on re-runs to avoid OAEP decryption errors
Persist auth ZooKeeper storage so SCRAM credentials survive broker restarts
Use stable MongoDB database name for local and codespace runs
Refactor setup step loop to early exits, skip metadata deploy when disabled
Fix CTST test runner to read vault admin credentials from cluster secrets

Instead of pruning all unused images (which destroys unrelated local images), remove only the CTST image after loading it into kind. The e2e image was already cleaned up with docker rmi.

Replace kubectl create with create --dry-run=client | kubectl apply or kubectl apply -f so that re-running the setup scripts updates existing resources instead of failing.

Replace the `sed -i 's/sorbet-data/miria-data/g'` workaround with the new PVC_NAME env var from zenko-operator's deploy-sorbet-resources.sh. Both deploys now explicitly specify their PVC_NAME, making the coupling with run-e2e-ctst.sh's claimName visible at a glance. See: scality/zenko-operator#599 Issue: ZKOP-530

Preserve ZENKO_MONGODB_DATABASE if already set instead of always generating a UUID, ensuring re-runs reuse the same database.

Tilt rebuilds the operator image from source, so pre-pulling the release image is redundant.

Skip network round-trip for helm repos already registered. Clone kafka-operator to a persistent directory to avoid re-cloning on every run.

… CRDs Strip stale metadata when copying root-ca secret to avoid resourceVersion conflict. Use kubectl apply --server-side for prometheus and kafka CRDs instead of create-or-replace pattern.

Refactor the action.yaml step loop from a big if block to early exit guards for clarity. Skip deploy-metadata.sh when ring tests are disabled.

Instead of regenerating the keypair secret on every run, check if it already exists and extract the public key from it. This prevents OAEP decryption errors caused by location configs encrypted with a previous keypair.

SCRAM-SHA-512 user registrations are stored in ZooKeeper. The auth ZooKeeper clone was using ephemeral storage, so credentials were lost on pod restart, causing notification-processor-destination4 to fail with SASL authentication errors.

Hardcoded credentials don't survive vault restarts. Read the vault admin key from the management-vault-admin-creds secret instead, matching how the CI run-e2e-ctst.sh script works. Also fix the account name to zenko-ctst to match CI.

bert-e · 2026-03-25T17:50:44Z

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-03-25T17:50:47Z

Missing Jira issue

It is mandatory to specify a Jira issue in the source branch name
in order to merge into development/2.14. The issue key must follow
the prefix of the branch.

I could not find such an issue in improvement/local-dev-idempotency.

delthas · 2026-03-25T17:57:10Z

Oops this was auto-created. Sorry. 😅

delthas added 11 commits March 25, 2026 18:48

setup: replace docker image prune -af with targeted rmi

362a3e8

Instead of pruning all unused images (which destroys unrelated local images), remove only the CTST image after loading it into kind. The e2e image was already cleaned up with docker rmi.

scripts: make kubectl create calls idempotent for re-runs

6832011

Replace kubectl create with create --dry-run=client | kubectl apply or kubectl apply -f so that re-running the setup scripts updates existing resources instead of failing.

setup: use stable MongoDB database name for local and codespace runs

0867dd1

Preserve ZENKO_MONGODB_DATABASE if already set instead of always generating a UUID, ensuring re-runs reuse the same database.

deploy-zkop: remove unnecessary operator image pull

02c5748

Tilt rebuilds the operator image from source, so pre-pulling the release image is redundant.

install-kind-deps: skip helm repo add if exists, persist kafka clone

e016206

Skip network round-trip for helm repos already registered. Clone kafka-operator to a persistent directory to avoid re-cloning on every run.

install-kind-deps: fix re-run failures for root-ca, prometheus, kafka…

3cefcaf

… CRDs Strip stale metadata when copying root-ca secret to avoid resourceVersion conflict. Use kubectl apply --server-side for prometheus and kafka CRDs instead of create-or-replace pattern.

setup: refactor step loop to early exits, skip metadata deploy

52ca8e5

Refactor the action.yaml step loop from a big if block to early exit guards for clarity. Skip deploy-metadata.sh when ring tests are disabled.

deploy-zenko: reuse existing keypair on re-runs

df20635

Instead of regenerating the keypair secret on every run, check if it already exists and extract the public key from it. This prevents OAEP decryption errors caused by location configs encrypted with a previous keypair.

delthas closed this Mar 25, 2026

delthas deleted the improvement/local-dev-idempotency branch March 25, 2026 21:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

e2e scripts: make setup idempotent and fix test runner credentials#2363

e2e scripts: make setup idempotent and fix test runner credentials#2363
delthas wants to merge 11 commits intodevelopment/2.14from
improvement/local-dev-idempotency

delthas commented Mar 25, 2026

Uh oh!

bert-e commented Mar 25, 2026

Uh oh!

bert-e commented Mar 25, 2026

Uh oh!

delthas commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

delthas commented Mar 25, 2026

Summary

Uh oh!

bert-e commented Mar 25, 2026

Hello delthas,

Uh oh!

bert-e commented Mar 25, 2026

Missing Jira issue

Uh oh!

delthas commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants