Flake: ScyllaCluster [It] should connect to cluster via Ingresses

[mflendrich]

Issue originally authored by tnozicka as #1476

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1456/pull-scylla-operator-e2e-gke-parallel/1712825884106297344#1:test-build-log.txt%3A1394

 • [FAILED] [493.231 seconds]
ScyllaCluster [It] should connect to cluster via Ingresses
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_expose.go:47
  Timeline >>
  STEP: Creating a new namespace @ 10/13/23 13:57:27.987
  Oct 13 13:57:28.126: INFO: Created namespace "e2e-test-scyllacluster-25s7q-rxxkg".
  STEP: Waiting for service account token Secret "e2e-user-token" in namespace "e2e-test-scyllacluster-25s7q-rxxkg". @ 10/13/23 13:57:28.312
  STEP: Waiting for default ServiceAccount in namespace "e2e-test-scyllacluster-25s7q-rxxkg". @ 10/13/23 13:57:28.962
  STEP: Creating a ScyllaCluster @ 10/13/23 13:57:29.122
  STEP: Waiting for ScyllaCluster to deploy @ 10/13/23 13:57:29.227
  Oct 13 14:00:25.783: INFO: ScyllaCluster e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd (RV=10407) is rolled out
  STEP: Verifying the ScyllaCluster @ 10/13/23 14:00:25.783
  Oct 13 14:00:25.833: INFO: Found 1 pvc(s) in namespace "e2e-test-scyllacluster-25s7q-rxxkg"
  Oct 13 14:00:25.833: INFO: Found 1 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd"
  STEP: Nodes reloaded the certificate @ 10/13/23 14:00:26.024
  STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 10/13/23 14:00:26.024
  Oct 13 14:00:26.150: INFO: ScyllaDB nodes have reached status consistency.
  STEP: Connecting via e2e-test-scyllacluster-25s7q-rxxkg.private.nodes.scylladb.com domain @ 10/13/23 14:00:26.22
  STEP: Injecting ingress controller address into the CQL connection bundle @ 10/13/23 14:00:26.22
  STEP: Saving CQL Connection Config in /tmp/connection-bundle-e2e-test-scyllacluster-25s7q-rxxkg-3188734847/e2e-test-scyllacluster-25s7q-rxxkg.private.nodes.scylladb.com.yaml @ 10/13/23 14:00:26.234
  STEP: Connecting to cluster via Ingress @ 10/13/23 14:00:26.234
  [FAILED] in [It] - github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_expose.go:132 @ 10/13/23 14:00:29.692
  STEP: Collecting events from namespace "e2e-test-scyllacluster-25s7q-rxxkg". @ 10/13/23 14:00:29.693
  STEP: Found 39 events. @ 10/13/23 14:00:29.708
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:57:30 +0000 UTC - event for basic-mh9kd-auth-token: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-auth-token created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:57:30 +0000 UTC - event for basic-mh9kd-member: {scyllacluster-controller } RoleBindingCreated: RoleBinding e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-member created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:57:30 +0000 UTC - event for basic-mh9kd-member: {scyllacluster-controller } ServiceAccountCreated: ServiceAccount e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-member created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:57:45 +0000 UTC - event for basic-mh9kd-local-client-ca: {scyllacluster-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-local-client-ca created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:57:45 +0000 UTC - event for basic-mh9kd-local-client-ca: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-local-client-ca created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:58:17 +0000 UTC - event for basic-mh9kd-local-user-admin: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-local-user-admin created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:58:44 +0000 UTC - event for basic-mh9kd-local-serving-ca: {scyllacluster-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-local-serving-ca created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:58:44 +0000 UTC - event for basic-mh9kd-local-serving-ca: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-local-serving-ca created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:14 +0000 UTC - event for basic-mh9kd-local-serving-certs: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-local-serving-certs created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:14 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-mh9kd-us-east-1-us-east-1a-0 Pod basic-mh9kd-us-east-1-us-east-1a-0 in StatefulSet basic-mh9kd-us-east-1-us-east-1a success
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:14 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-mh9kd-us-east-1-us-east-1a-0 in StatefulSet basic-mh9kd-us-east-1-us-east-1a successful
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:14 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a: {scyllacluster-controller } StatefulSetCreated: StatefulSet e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-us-east-1-us-east-1a created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:14 +0000 UTC - event for data-basic-mh9kd-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-gvzmp_d7b16ac4-6b6d-40d9-a10e-cbca1aacd04f } ProvisioningSucceeded: Successfully provisioned volume pvc-809c57e8-751c-4808-b435-b7567afc4390
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:14 +0000 UTC - event for data-basic-mh9kd-us-east-1-us-east-1a-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "local.csi.scylladb.com" or manually created by system administrator
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:14 +0000 UTC - event for data-basic-mh9kd-us-east-1-us-east-1a-0: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:14 +0000 UTC - event for data-basic-mh9kd-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-gvzmp_d7b16ac4-6b6d-40d9-a10e-cbca1aacd04f } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-25s7q-rxxkg/data-basic-mh9kd-us-east-1-us-east-1a-0"
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:15 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-us-east-1-us-east-1a-0 to gke-so-3e139581-9dcd-4961-acd-workers-2ab0b6cb-7r2q
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:15 +0000 UTC - event for nodeconfig-podinfo-6a5c6c7a-4b01-44f8-9db3-26c27aaf625f: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-25s7q-rxxkg/nodeconfig-podinfo-6a5c6c7a-4b01-44f8-9db3-26c27aaf625f created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:16 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {kubelet gke-so-3e139581-9dcd-4961-acd-workers-2ab0b6cb-7r2q} Created: Created container sidecar-injection
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:16 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {kubelet gke-so-3e139581-9dcd-4961-acd-workers-2ab0b6cb-7r2q} Started: Started container sidecar-injection
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:16 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {kubelet gke-so-3e139581-9dcd-4961-acd-workers-2ab0b6cb-7r2q} Pulled: Container image "quay.io/scylladb-dev/ci@sha256:6de2a72da0b02b6d25df360ab418ca379679b4b3ed450b5b935d2fea9edccd9f" already present on machine
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:24 +0000 UTC - event for basic-mh9kd: {scyllacluster-controller } PodDisruptionBudgetCreated: PodDisruptionBudget e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:24 +0000 UTC - event for basic-mh9kd-client: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-client created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:24 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {kubelet gke-so-3e139581-9dcd-4961-acd-workers-2ab0b6cb-7r2q} Pulling: Pulling image "docker.io/scylladb/scylla-manager-agent:3.1.0"
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:24 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-us-east-1-us-east-1a-0 created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:24 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {kubelet gke-so-3e139581-9dcd-4961-acd-workers-2ab0b6cb-7r2q} Pulled: Container image "docker.io/scylladb/scylla:5.2.7" already present on machine
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:24 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {kubelet gke-so-3e139581-9dcd-4961-acd-workers-2ab0b6cb-7r2q} Started: Started container scylla
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:24 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {kubelet gke-so-3e139581-9dcd-4961-acd-workers-2ab0b6cb-7r2q} Created: Created container scylla
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:27 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {kubelet gke-so-3e139581-9dcd-4961-acd-workers-2ab0b6cb-7r2q} Created: Created container scylla-manager-agent
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:27 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {kubelet gke-so-3e139581-9dcd-4961-acd-workers-2ab0b6cb-7r2q} Pulled: Successfully pulled image "docker.io/scylladb/scylla-manager-agent:3.1.0" in 3.0291668s (3.0291847s including waiting)
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:28 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {kubelet gke-so-3e139581-9dcd-4961-acd-workers-2ab0b6cb-7r2q} Started: Started container scylla-manager-agent
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:28 +0000 UTC - event for nodeconfig-podinfo-6a5c6c7a-4b01-44f8-9db3-26c27aaf625f: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-25s7q-rxxkg/nodeconfig-podinfo-6a5c6c7a-4b01-44f8-9db3-26c27aaf625f updated
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:35 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {kubelet gke-so-3e139581-9dcd-4961-acd-workers-2ab0b6cb-7r2q} Unhealthy: Startup probe failed: HTTP probe failed with statuscode: 503
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:45 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {kubelet gke-so-3e139581-9dcd-4961-acd-workers-2ab0b6cb-7r2q} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:48 +0000 UTC - event for basic-mh9kd-local-cql-connection-configs-admin: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-local-cql-connection-configs-admin created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:48 +0000 UTC - event for basic-mh9kd-local-serving-certs: {scyllacluster-controller } SecretUpdated: Secret e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-local-serving-certs updated
  Oct 13 14:00:29.708: INFO: At 2023-10-13 13:59:58 +0000 UTC - event for basic-mh9kd-client-cql: {scyllacluster-controller } IngressCreated: Ingress e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-client-cql created
  Oct 13 14:00:29.708: INFO: At 2023-10-13 14:00:25 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0: {scyllacluster-controller } ServiceUpdated: Service e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-us-east-1-us-east-1a-0 updated
  Oct 13 14:00:29.708: INFO: At 2023-10-13 14:00:25 +0000 UTC - event for basic-mh9kd-us-east-1-us-east-1a-0-cql: {scyllacluster-controller } IngressCreated: Ingress e2e-test-scyllacluster-25s7q-rxxkg/basic-mh9kd-us-east-1-us-east-1a-0-cql created
  STEP: Collecting dumps from namespace "e2e-test-scyllacluster-25s7q-rxxkg". @ 10/13/23 14:00:29.708
  STEP: Destroying namespace "e2e-test-scyllacluster-25s7q-rxxkg". @ 10/13/23 14:00:30.711
  STEP: Waiting for namespace "e2e-test-scyllacluster-25s7q-rxxkg" to be removed. @ 10/13/23 14:00:30.728
  << Timeline
  [FAILED] Unexpected error:
      <*errors.errorString | 0x34aa230>: 
      no connections were made when creating the session
      {
          s: "no connections were made when creating the session",
      }
  occurred
  In [It] at: github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_expose.go:132 @ 10/13/23 14:00:29.692
  Full Stack Trace
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.glob..func4.1()
    	github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_expose.go:132 +0x1123 

[scylla-operator-bot]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out

/lifecycle rotten

[scylla-operator-bot]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out

/close not-planned

[scylla-operator-bot]

@scylla-operator-bot[bot]: Closing this issue, marking it as "Not Planned".

In response to this:

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.