Run cucumber tests out of cluster by SylvainSenechal · Pull Request #2372 · scality/Zenko

SylvainSenechal · 2026-04-07T11:50:49Z

-Cucumber tests are run directly in the github runner, no need for a pod anymore. This is quite nice for codespace testing where I got the opportunity to test it as I was making the pr and fixing the few bugs that didn't work

Cleanup a few unused files, related to this pr and also a few unrelated
updated readme to explain how to run the tests
fixed a bug in one of the bash setup file that was breaking one of the zenko tests
fixed an issue with issue location creation, stopping the codespace from being setup properly because one of the gcp location env var isn't defined in codespace, so we were creating an empty location

Future work :

Some cleanup around env variables vs world parameters, its a bit messy now, we could get rid of ctst world parameters and just use env variables everywhere
Unify tests (zenko mocha, cucumber) setup, unify tests package under the same js project.

Once these 2 are done, I believe it will be much easier to work on other tasks, especially everything dealing with the setup in the bash and python files

bert-e · 2026-04-07T11:50:55Z

Hello sylvainsenechal,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-04-07T11:51:02Z

Jira issue not found

The Jira issue ZENKO-5209 was not found.

SylvainSenechal · 2026-04-10T16:24:23Z

Not a big fan of this file, quite long, with a flag depending if it runs for ctst or zenko tests.
In a next pr i wanna refine it to make it work for all tests without flags

I also wanna do something about world parameters vs env variables.

Currently for cucumber tests, what we have is :

Env var XXX is defined in end2end.yaml

In this file, we create a world parameters thats Xxx=XXX

Cucumber tests uses this.parameters.Xxx

We could remove one level of indirection and directly use env variables in cucumber tests.
This would also be helpful to track env variable utilisation better, because here the world parameters are sometimes renamed, example :

KeycloakUsername = $OIDC_USERNAME

So when grepping OIDC_USERNAME, you can't directly see that it's used because the test uses this.parameters.KeycloakUsername

SylvainSenechal · 2026-04-10T16:26:26Z

Not directly related to this pr but didn't want to create a ticket just for that :
azure/docker-entrypoint and the following dockerfile aren't used, what's used is the azure-mock.yaml file where the image is directly defined inside of it.

Btw, in the deleted dockerfile, there is a typo in the azurite version with an extra whitespace, so not even sure it was working before 🧐

SylvainSenechal · 2026-04-10T16:27:05Z

  containers:
  - image: ghcr.io/scality/cloudserver:8.8.59
    name: aws-mock
-    ports:


defined again line 83

SylvainSenechal · 2026-04-10T16:35:34Z

 };

 type VolumeGetConfig = {
-    targetZenkoKubeconfigPath?: string;


These are translated into cli commands for drctlt, and they were misnamed

bert-e · 2026-04-10T16:37:57Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

francoisferrand · 2026-04-14T06:47:41Z


-After deployment is done, which you can follow by opening another terminal, you will be able to access S3 service through a port-forward.
-First find a cloudserver connector using the following command:
+After deployment is done, the devcontainer setup configures ingress endpoints and `/etc/hosts` entries


nit: not for now, but using /etc/hosts is probably a nogo for local development... so we'll eventually need to come back to this

francoisferrand · 2026-04-14T06:49:03Z


 ```bash
-    aws s3 ls --endpoint http://localhost:8080
+aws s3 ls --endpoint http://s3.zenko.local


endpoint can also be set in configuration. For exemple in my .aws/config I have defined a scaleway profile:

[profile scaleway] endpoint_url = https://s3.fr-par.scw.cloud region = fr-par

I moved the endpoint to the "configure set" commands. Anyways i dont think this is used much, i would hope that with codespace we can write a test and run it to test it, including sdk commands

francoisferrand · 2026-04-14T06:49:52Z

+export ACCESS_KEY=$(kubectl get secret end2end-management-vault-admin-creds.v1 -o jsonpath='{.data.accessKey}' | base64 -d)
+export SECRET_KEY=$(kubectl get secret end2end-management-vault-admin-creds.v1  -o jsonpath='{.data.secretKey}' | base64 -d)
 ```

-Then configure aws cli with the following command
+Configure the AWS CLI:

 ```bash
-    aws configure set aws_access_key_id $ACCESS_KEY
-    aws configure set aws_secret_access_key $SECRET_KEY
-    aws configure set region us-east-1
+aws configure set aws_access_key_id $ACCESS_KEY
+aws configure set aws_secret_access_key $SECRET_KEY
+aws configure set region us-east-1
 ```


nit: should this be done by setup-e2e-env ?

Could be but I think no need, I talked about it a bit in the other comment, but I think this should barely be used, as I would expect people to just develop a test and run the test directly, so don't really need to run s3 cli commands here. And setup-e2e-env is more for things that could be used both for codespace and github ci, than only for codespace

francoisferrand · 2026-04-14T07:10:35Z

+    // which would hang when called from Node.js exec() whose stdin pipe is
+    // never closed by the caller.
+    return execShellCommand(
+        `kubectl run ${podName} --image=busybox:1 --attach --rm --restart=Never --overrides='${spec}'`,


not the way to do this: should be done with kubectl debug instead

what's the problem with kubectl run 🤔 The pod stays there after the test is run ?

not a "problem" with kubectl run, just not the same use : kubectl run is designed to run a "workload" (i.e. some production stuff) while kubectl debug is designed to do exactly what you want (i.e. like running a command in another pod -even if it does not have a shell-, ...)

sure it can work -especially in CI- with kubectl run, but the future is to use kubectl debug (instead of kubectl run/kuebctl exec) for most troubleshooting procedures for exemple

in particular kubectl debug knows how to "share" volumes already, no need to add coupling on the way the pod is started

delthas · 2026-04-15T15:25:15Z

    local name="$1"
    local host="$2"
    local service="$3"
+    local port="${4:-name: http}"


Would it perhaps be nicer to have this be always a number and have number: ${port} below? Then we could just use 80 as the default instead of name: http

ah ok yeah, i didnt know we could use name: http syntax when the port is 80

francoisferrand · 2026-04-15T16:59:09Z

  TMATE_SERVER_ED25519_FINGERPRINT: ${{ secrets.TMATE_SERVER_ED25519_FINGERPRINT }}
  # Mocha reporter configuration
-  MOCHA_FILE: /reports/test-results-[hash].xml
+  MOCHA_FILE: reports/test-results-[hash].xml


no correct, properly fixed in #2375 → to be removed from here
(and rebased on top of it, hopefully it will merge in a few minutes...)

francoisferrand · 2026-04-15T17:03:55Z

+    # PRA admin credentials (may not exist for non-PRA runs; ignore errors)
+    export ADMIN_PRA_ACCESS_KEY_ID=$(kubectl get secret ${ZENKO_NAME}-pra-management-vault-admin-creds.v1 -o jsonpath='{.data.accessKey}' 2>/dev/null | base64 -d 2>/dev/null || echo "")
+    export ADMIN_PRA_SECRET_ACCESS_KEY=$(kubectl get secret ${ZENKO_NAME}-pra-management-vault-admin-creds.v1 -o jsonpath='{.data.secretKey}' 2>/dev/null | base64 -d 2>/dev/null || echo "")
+
+    # --- 11. Service user credentials ---
+    BACKBEAT_LCBP_1_CREDS=$(kubectl get secret -l app.kubernetes.io/name=backbeat-lcbp-user-creds,app.kubernetes.io/instance=${ZENKO_NAME} -o jsonpath='{.items[0].data.backbeat-lifecycle-bp-1\.json}' | base64 -d)
+    BACKBEAT_LCC_1_CREDS=$(kubectl get secret -l app.kubernetes.io/name=backbeat-lcc-user-creds,app.kubernetes.io/instance=${ZENKO_NAME} -o jsonpath='{.items[0].data.backbeat-lifecycle-conductor-1\.json}' | base64 -d)
+    BACKBEAT_LCOP_1_CREDS=$(kubectl get secret -l app.kubernetes.io/name=backbeat-lcop-user-creds,app.kubernetes.io/instance=${ZENKO_NAME} -o jsonpath='{.items[0].data.backbeat-lifecycle-op-1\.json}' | base64 -d)
+    BACKBEAT_QP_1_CREDS=$(kubectl get secret -l app.kubernetes.io/name=backbeat-qp-user-creds,app.kubernetes.io/instance=${ZENKO_NAME} -o jsonpath='{.items[0].data.backbeat-qp-1\.json}' | base64 -d)
+    SORBET_FWD_2_ACCESSKEY=$(kubectl get secret -l app.kubernetes.io/name=sorbet-fwd-creds,app.kubernetes.io/instance=${ZENKO_NAME} -o jsonpath='{.items[0].data.accessKey}' | base64 -d)
+    SORBET_FWD_2_SECRETKEY=$(kubectl get secret -l app.kubernetes.io/name=sorbet-fwd-creds,app.kubernetes.io/instance=${ZENKO_NAME} -o jsonpath='{.items[0].data.secretKey}' | base64 -d)
+    export SERVICE_USERS_CREDENTIALS=$(echo '{"backbeat-lifecycle-bp-1":'"${BACKBEAT_LCBP_1_CREDS}"',"backbeat-lifecycle-conductor-1":'"${BACKBEAT_LCC_1_CREDS}"',"backbeat-lifecycle-op-1":'"${BACKBEAT_LCOP_1_CREDS}"',"backbeat-qp-1":'"${BACKBEAT_QP_1_CREDS}"',"sorbet-fwd-2":{"accessKey":"'"${SORBET_FWD_2_ACCESSKEY}"'","secretKey":"'"${SORBET_FWD_2_SECRETKEY}"'"}}' | jq -R)
+
+    # --- 12. Kafka topics for sorbet ---
+    SORBET_CONFIG=$(kubectl get secret -l app.kubernetes.io/name=cold-sorbet-config-e2e-azure-archive,app.kubernetes.io/instance=${ZENKO_NAME} \
+        -o jsonpath='{.items[0].data.config\.json}' | base64 -di)
+    export KAFKA_DEAD_LETTER_TOPIC=$(echo "${SORBET_CONFIG}" | jq -r '."kafka-dead-letter-topic"')
+    export KAFKA_OBJECT_TASK_TOPIC=$(echo "${SORBET_CONFIG}" | jq -r '."kafka-object-task-topic"')
+    export KAFKA_GC_REQUEST_TOPIC=$(echo "${SORBET_CONFIG}" | jq -r '."kafka-gc-request-topic"')


for followup: everything we retrieve from secrets/k8s can be retrieved directly from CTST. Just needs creds and ZENKO_NAME

francoisferrand · 2026-04-15T17:13:28Z

+        _KAFKA_PF_PID=$!
+        timeout 10 bash -c "until ss -tlnp 2>/dev/null | grep -q ':${KAFKA_PORT}'; do sleep 0.2; done"
+    fi
+    export KAFKA_HOST_PORT="localhost:${KAFKA_PORT}"


another idea for followup: instead of port-forwarding here, maybe we can port-forward directly in CTST or run a command in the k8s cluster (depending on the case)

this would have the same benefit as above: less setup, tests can autonomously do what they need, as long as they know how to access zenko running in k8s → fewer variables

ah you mean directly in the code 🤔 I haven't thought of that, it's probably possible to do

francoisferrand · 2026-04-15T17:16:53Z

    printf 'CRR_SOURCE_INFO<<EOF\n%s\nEOF\n' "$CRR_SOURCE_INFO" >> "$GITHUB_ENV"
    printf 'CRR_DESTINATION_INFO<<EOF\n%s\nEOF\n' "$CRR_DESTINATION_INFO" >> "$GITHUB_ENV"
+    # CTST-specific vars (only if CTST setup ran)
+    if [ "${SKIP_CTST:-}" != "1" ]; then


can't we move this inside of the existing SKIP_CTST block ?

(and wondering why we need these, even though we generate a "world" file which should contain everything needed by ctst... probably something to further cleanup there, eventually)

Actually yeah since its already in world parameters, no need to keep it

francoisferrand · 2026-04-15T17:18:04Z

+Then('prometheus should scrap federated metrics from DR sink', { timeout: 180000 }, async () => {
    const prom = new PrometheusDriver({
-        endpoint: `http://${this.parameters.PrometheusService}:9090`,
+        endpoint: 'http://localhost:9090',


should keep this.parameters.PrometheusService here : and just set the parameter to localhost.

francoisferrand · 2026-04-15T17:20:01Z

 import Zenko from 'world/Zenko';

+// The mock-sorbet pod uses a distroless image (no shell, rm, find, etc.),
+// so we cannot kubectl exec into it.  Instead, we spin up a short-lived


more precisely, we can kubectl exec (for exemple kubectl exec -- sorbetctl ...): but we cannot exec to run a shell or shell commands indeed

…ace, causing codespace setup failures Issue: ZENKO-5209

Issue: ZENKO-5209

…ber tests Issue: ZENKO-5209

…zenko cluster Issue: ZENKO-5209

…r tests following new method to run cucumber tests out of the zenko cluster Issue: ZENKO-5209

Issue: ZENKO-5209

SylvainSenechal · 2026-04-16T23:32:33Z

/approve

bert-e · 2026-04-17T02:05:05Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.14

The following branches have NOT changed:

development/2.10
development/2.11
development/2.12
development/2.13
development/2.5
development/2.6
development/2.7
development/2.8
development/2.9

Please check the status of the associated issue ZENKO-5209.

Goodbye sylvainsenechal.

The following options are set: approve

SylvainSenechal force-pushed the improvement/ZENKO-5209 branch 7 times, most recently from d4dac97 to f3e4d82 Compare April 10, 2026 15:57

SylvainSenechal commented Apr 10, 2026

View reviewed changes

Comment thread .github/workflows/end2end.yaml

SylvainSenechal commented Apr 10, 2026

View reviewed changes

SylvainSenechal marked this pull request as ready for review April 10, 2026 16:37

scality deleted a comment from bert-e Apr 10, 2026

SylvainSenechal requested review from a team, benzekrimaha and delthas April 10, 2026 16:43

SylvainSenechal force-pushed the improvement/ZENKO-5209 branch 2 times, most recently from e2cc589 to 2acec2d Compare April 13, 2026 15:11

francoisferrand reviewed Apr 14, 2026

View reviewed changes

SylvainSenechal force-pushed the improvement/ZENKO-5209 branch 2 times, most recently from f3ec7ad to c55b40f Compare April 14, 2026 16:54

SylvainSenechal requested a review from francoisferrand April 14, 2026 16:54

delthas approved these changes Apr 15, 2026

View reviewed changes

francoisferrand reviewed Apr 15, 2026

View reviewed changes

francoisferrand approved these changes Apr 15, 2026

View reviewed changes

SylvainSenechal force-pushed the improvement/ZENKO-5209 branch 2 times, most recently from 33fb80e to 5f421ed Compare April 15, 2026 22:15

benzekrimaha approved these changes Apr 16, 2026

View reviewed changes

SylvainSenechal force-pushed the improvement/ZENKO-5209 branch from 5f421ed to 093e162 Compare April 16, 2026 16:32

SylvainSenechal added 8 commits April 16, 2026 23:32

skip empty location creation as they are not always defined in codesp…

56f232c

…ace, causing codespace setup failures Issue: ZENKO-5209

remove unused azure mock dockerfile and entrypoint

95d2cae

Issue: ZENKO-5209

fix aws mock yaml file double port declaration

a32eddc

Issue: ZENKO-5209

update test code to run cucumber tests out of the zenko cluster

53896d1

Issue: ZENKO-5209

update codespace devcontainer setup following new method to run cucum…

d4d1ff4

…ber tests Issue: ZENKO-5209

update readmes following new method to run cucumber tests out of the …

6a86e77

…zenko cluster Issue: ZENKO-5209

remove useless cucumber test dockerfile and old helper to run cucumbe…

9baff3e

…r tests following new method to run cucumber tests out of the zenko cluster Issue: ZENKO-5209

add yarn build after yarn lint step, fix errors found by yarn build

602bdc4

Issue: ZENKO-5209

SylvainSenechal force-pushed the improvement/ZENKO-5209 branch from 093e162 to 602bdc4 Compare April 16, 2026 23:32

bert-e merged commit 602bdc4 into development/2.14 Apr 17, 2026
14 checks passed

bert-e deleted the improvement/ZENKO-5209 branch April 17, 2026 02:05

Conversation

SylvainSenechal commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bert-e commented Apr 7, 2026

Hello sylvainsenechal,

Uh oh!

bert-e commented Apr 7, 2026

Jira issue not found

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bert-e commented Apr 10, 2026

Waiting for approval

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal commented Apr 16, 2026

Uh oh!

bert-e commented Apr 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

SylvainSenechal commented Apr 7, 2026 •

edited

Loading

francoisferrand Apr 15, 2026 •

edited

Loading

francoisferrand Apr 15, 2026 •

edited

Loading