Open
Description
Link to the job that flaked.
Snippet of what failed.
ScyllaCluster [It] should setup and maintain up to date TLS certificates
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:42
Timeline >>
STEP: Creating a new namespace @ 08/26/24 21:30:43.436
Aug 26 21:30:43.466: INFO: Created namespace "e2e-test-scyllacluster-2ms98-0-85njt".
STEP: Waiting for service account token Secret "e2e-user-token" in namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:30:43.522
STEP: Waiting for default ServiceAccount in namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:30:44.178
STEP: Waiting for kube-root-ca.crt in namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:30:44.492
STEP: Creating an initial ScyllaCluster with a single node @ 08/26/24 21:30:44.502
STEP: Waiting for the ScyllaCluster to roll out (RV=5639) @ 08/26/24 21:30:44.533
Aug 26 21:35:13.491: INFO: ScyllaCluster e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 (RV=11632) is rolled out
STEP: Verifying the ScyllaCluster @ 08/26/24 21:35:13.491
Aug 26 21:35:13.648: INFO: Found 1 pvc(s) in namespace "e2e-test-scyllacluster-2ms98-0-85njt"
Aug 26 21:35:13.648: INFO: Found 1 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8"
STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 08/26/24 21:35:13.757
Aug 26 21:35:14.163: INFO: ScyllaDB nodes have reached status consistency.
STEP: Inserting data @ 08/26/24 21:35:14.259
Aug 26 21:35:14.259: INFO: Creating CQL session (hosts="10.17.250.51")
STEP: Inserting data @ 08/26/24 21:35:14.263
Aug 26 21:35:14.263: INFO: Creating keyspace "6pztcv98" with RF "'replication_factor': 1"
Aug 26 21:35:14.271: INFO: Creating table "6pztcv98"."test"
Aug 26 21:35:14.281: INFO: Inserting data into table "6pztcv98"."test"
Aug 26 21:35:14.296: INFO: Awaiting schema agreement
Aug 26 21:35:14.297: INFO: Schema agreement reached
STEP: Verifying the data @ 08/26/24 21:35:14.297
Aug 26 21:35:14.297: INFO: Reading data from table "6pztcv98"."test"
STEP: Scaling the ScyllaCluster to 1 replicas setting domains to [] @ 08/26/24 21:35:14.3
STEP: Waiting for the ScyllaCluster to roll out (RV=11670) @ 08/26/24 21:35:14.496
Aug 26 21:35:14.643: INFO: ScyllaCluster e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 (RV=11670) is rolled out
STEP: Verifying the ScyllaCluster @ 08/26/24 21:35:14.648
Aug 26 21:35:14.905: INFO: Found 1 pvc(s) in namespace "e2e-test-scyllacluster-2ms98-0-85njt"
Aug 26 21:35:14.905: INFO: Found 1 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8"
STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 08/26/24 21:35:14.981
Aug 26 21:35:15.253: INFO: ScyllaDB nodes have reached status consistency.
STEP: Verifying TLS API objects @ 08/26/24 21:35:15.326
STEP: Verifying certificates @ 08/26/24 21:35:15.384
Aug 26 21:35:15.456: INFO: Starting to probe node "10.17.250.51" for correct certs
Aug 26 21:35:15.530: INFO: Node "10.17.250.51" has correct certs
STEP: Scaling the ScyllaCluster to 1 replicas setting domains to ["foo.scylladb.com" "bar.scylladb.com"] @ 08/26/24 21:35:15.53
STEP: Waiting for the ScyllaCluster to roll out (RV=11738) @ 08/26/24 21:35:15.601
Aug 26 21:35:16.039: INFO: ScyllaCluster e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 (RV=11775) is rolled out
STEP: Verifying the ScyllaCluster @ 08/26/24 21:35:16.039
Aug 26 21:35:16.296: INFO: Found 1 pvc(s) in namespace "e2e-test-scyllacluster-2ms98-0-85njt"
Aug 26 21:35:16.296: INFO: Found 1 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8"
STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 08/26/24 21:35:16.458
Aug 26 21:35:16.644: INFO: ScyllaDB nodes have reached status consistency.
STEP: Verifying TLS API objects @ 08/26/24 21:35:16.702
STEP: Verifying certificates @ 08/26/24 21:35:16.856
Aug 26 21:35:16.927: INFO: Starting to probe node "10.17.250.51" for correct certs
Aug 26 21:36:11.572: INFO: Node "10.17.250.51" has correct certs
STEP: Scaling the ScyllaCluster to 2 replicas setting domains to ["foo.scylladb.com" "bar.scylladb.com"] @ 08/26/24 21:36:11.572
STEP: Waiting for the ScyllaCluster to roll out (RV=13780) @ 08/26/24 21:36:11.604
Aug 26 21:37:44.388: INFO: ScyllaCluster e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 (RV=16384) is rolled out
STEP: Verifying the ScyllaCluster @ 08/26/24 21:37:44.389
Aug 26 21:37:44.431: INFO: Found 2 pvc(s) in namespace "e2e-test-scyllacluster-2ms98-0-85njt"
Aug 26 21:37:44.431: INFO: Found 2 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8"
STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 08/26/24 21:37:44.496
Aug 26 21:37:44.576: INFO: ScyllaDB nodes have reached status consistency.
STEP: Verifying TLS API objects @ 08/26/24 21:37:44.608
STEP: Verifying certificates @ 08/26/24 21:37:44.654
[FAILED] in [It] - github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:211 @ 08/26/24 21:37:44.68
STEP: Collecting events from namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:37:44.68
STEP: Found 85 events. @ 08/26/24 21:37:44.694
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:30:48 +0000 UTC - event for basic-nztd8: {scyllaclustermigration-controller } ScyllaDBDatacenterCreated: ScyllaDBDatacenter e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:30:48 +0000 UTC - event for basic-nztd8-member: {scylladbdatacenter-controller } ServiceAccountCreated: ServiceAccount e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-member created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:30:49 +0000 UTC - event for basic-nztd8-member: {scylladbdatacenter-controller } RoleBindingCreated: RoleBinding e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-member created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:30:50 +0000 UTC - event for basic-nztd8-auth-token: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-auth-token created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:31:05 +0000 UTC - event for basic-nztd8-local-client-ca: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-client-ca created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:31:05 +0000 UTC - event for basic-nztd8-local-client-ca: {scylladbdatacenter-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-client-ca created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:31:41 +0000 UTC - event for basic-nztd8-local-user-admin: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-user-admin created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:10 +0000 UTC - event for basic-nztd8-local-serving-ca: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-serving-ca created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:10 +0000 UTC - event for basic-nztd8-local-serving-ca: {scylladbdatacenter-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-serving-ca created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8: {scylladbdatacenter-controller } PodDisruptionBudgetCreated: PodDisruptionBudget e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8: {controllermanager } NoPods: No matching pods found
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8-client: {scylladbdatacenter-controller } ServiceCreated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-client created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8-managed-config: {scylladbdatacenter-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-managed-config created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {scylladbdatacenter-controller } ServiceCreated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-0 created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-local-cql-connection-configs-admin: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-cql-connection-configs-admin created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-local-serving-certs: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-serving-certs created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {scylladbdatacenter-controller } StatefulSetCreated: StatefulSet e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-nztd8-us-east-1-us-east-1a-0 in StatefulSet basic-nztd8-us-east-1-us-east-1a successful
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-nztd8-us-east-1-us-east-1a-0 Pod basic-nztd8-us-east-1-us-east-1a-0 in StatefulSet basic-nztd8-us-east-1-us-east-1a success
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-jrndw_b6271f4d-bd61-41ae-9a05-e05fa1264764 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-2ms98-0-85njt/data-basic-nztd8-us-east-1-us-east-1a-0"
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-0: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'local.csi.scylladb.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-0: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-jrndw_b6271f4d-bd61-41ae-9a05-e05fa1264764 } ProvisioningSucceeded: Successfully provisioned volume pvc-80a399a7-965f-4ccb-b26d-9a14946e88ea
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:21 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-0 to gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:21 +0000 UTC - event for nodeconfig-podinfo-26139921-f3df-4a43-995e-b9f665409456: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/nodeconfig-podinfo-26139921-f3df-4a43-995e-b9f665409456 created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:23 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:26 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container sidecar-injection
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:28 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container sidecar-injection
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:02 +0000 UTC - event for basic-nztd8-local-serving-certs: {scylladbdatacenter-controller } SecretUpdated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-serving-certs updated
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:15 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "docker.io/scylladb/scylla:6.1.0" already present on machine
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:16 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylla
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:17 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylladb-api-status-probe
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:17 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:17 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylla
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:19 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:19 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylladb-api-status-probe
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:23 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylladb-ignition
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:25 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylladb-ignition
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:25 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:3.3.0" already present on machine
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:25 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylla-manager-agent
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:28 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylla-manager-agent
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:32 +0000 UTC - event for nodeconfig-podinfo-26139921-f3df-4a43-995e-b9f665409456: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/nodeconfig-podinfo-26139921-f3df-4a43-995e-b9f665409456 updated
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:33 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: dial tcp 10.125.193.50:8080: connect: connection refused
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:33 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: Get "http://10.125.193.50:42081/readyz": dial tcp 10.125.193.50:42081: connect: connection refused
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:33 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: dial tcp 10.125.193.50:10001: connect: connection refused
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:41 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:41 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Startup probe failed: HTTP probe failed with statuscode: 503
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:52 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 500
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:35:13 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {scylladbdatacenter-controller } ServiceUpdated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-0 updated
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:35:15 +0000 UTC - event for basic-nztd8: {scyllaclustermigration-controller } ScyllaDBDatacenterUpdated: ScyllaDBDatacenter e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 updated
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:35:15 +0000 UTC - event for basic-nztd8-local-cql-connection-configs-admin: {scylladbdatacenter-controller } SecretUpdated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-cql-connection-configs-admin updated
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:11 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {scylladbdatacenter-controller } ServiceCreated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-1 created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-nztd8-us-east-1-us-east-1a-1 in StatefulSet basic-nztd8-us-east-1-us-east-1a successful
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-nztd8-us-east-1-us-east-1a-1 Pod basic-nztd8-us-east-1-us-east-1a-1 in StatefulSet basic-nztd8-us-east-1-us-east-1a success
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-1: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-1: {local.csi.scylladb.com_local-csi-driver-jrndw_b6271f4d-bd61-41ae-9a05-e05fa1264764 } ProvisioningSucceeded: Successfully provisioned volume pvc-ebb938df-3693-4f56-adb3-fc8ef5f6cb72
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-1: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'local.csi.scylladb.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-1: {local.csi.scylladb.com_local-csi-driver-jrndw_b6271f4d-bd61-41ae-9a05-e05fa1264764 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-2ms98-0-85njt/data-basic-nztd8-us-east-1-us-east-1a-1"
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:13 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-1 to gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:13 +0000 UTC - event for nodeconfig-podinfo-e3f5fefe-8131-43c6-872f-014e4043e82f: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/nodeconfig-podinfo-e3f5fefe-8131-43c6-872f-014e4043e82f created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:15 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container sidecar-injection
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:15 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:17 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container sidecar-injection
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:43 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "docker.io/scylladb/scylla:6.1.0" already present on machine
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:44 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylla
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:44 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:44 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylla
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:45 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylladb-api-status-probe
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:46 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:46 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylladb-api-status-probe
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:47 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylladb-ignition
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:48 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylladb-ignition
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:48 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylla-manager-agent
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:48 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:3.3.0" already present on machine
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:48 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylla-manager-agent
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:49 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: dial tcp 10.125.193.92:8080: connect: connection refused
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:49 +0000 UTC - event for nodeconfig-podinfo-e3f5fefe-8131-43c6-872f-014e4043e82f: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/nodeconfig-podinfo-e3f5fefe-8131-43c6-872f-014e4043e82f updated
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:50 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: dial tcp 10.125.193.92:10001: connect: connection refused
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:50 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: Get "http://10.125.193.92:42081/readyz": dial tcp 10.125.193.92:42081: connect: connection refused
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:34 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {scylladbdatacenter-controller } StatefulSetUpdated: StatefulSet e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a updated
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:34 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {scylladbdatacenter-controller } ServiceUpdated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-1 updated
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:44 +0000 UTC - event for basic-nztd8: {controllermanager } CalculateExpectedPodCountFailed: Failed to calculate the number of expected pods: jobs.batch does not implement the scale subresource
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:44 +0000 UTC - event for cleanup-basic-nztd8-us-east-1-us-east-1a-0: {scylladbdatacenter-controller } JobCreated: Job e2e-test-scyllacluster-2ms98-0-85njt/cleanup-basic-nztd8-us-east-1-us-east-1a-0 created
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:44 +0000 UTC - event for cleanup-basic-nztd8-us-east-1-us-east-1a-0: {job-controller } SuccessfulCreate: Created pod: cleanup-basic-nztd8-us-east-1-us-east-1a-0-v4lbp
Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:44 +0000 UTC - event for cleanup-basic-nztd8-us-east-1-us-east-1a-0-v4lbp: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-2ms98-0-85njt/cleanup-basic-nztd8-us-east-1-us-east-1a-0-v4lbp to gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl
STEP: Collecting dumps from namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:37:44.694
STEP: Destroying namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:37:45.437
STEP: Waiting for namespace "e2e-test-scyllacluster-2ms98-0-85njt" to be removed. @ 08/26/24 21:37:45.449
<< Timeline
[FAILED] Expected
<[]string | len:5, cap:5>: ["10.17.250.51", "10.17.255.223", "10.125.193.50", "10.125.193.92", ""]
to have length 4
In [It] at: github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:211 @ 08/26/24 21:37:44.68
Full Stack Trace
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.init.func16.1.1({{0xc000cbbdc0?, 0x1?, 0x0?}, 0xc000bd6708?}, 0xc000a2dcf0, 0xc0007c7500, {0x35a2038, 0xc0005c5810}, {0xc0007f3040, 0x1, ...})
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:211 +0x21c9
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.init.func16.1()
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:259 +0x930
Looks like a cleanup Pod - triggered after considering ScyllaCluster as rolled out (it's an expected edge case) - got picked up by the test logic and IP of it (empty one) failed the assertion.