Node setup fails on filesystem creation in `v1.32.3-gke.1170000`

[rzetelskik]

What happened?

Node setup fails on filesystem creation in GKE v1.32.3-gke.1170000.

E0409 10:57:50.806556       1 nodesetup/controller.go:192] "Unhandled Error" err="syncing key 'scylladb-nodepool-1' failed: [can't sync filesystems: failed to create filesystems: can't create filesystem \"xfs\" on device \"/dev/md/nvmes\" at \"/dev/md127\": can't run mkfs with args [-t xfs -b size=4096 -K /dev/md127]: exit status 1, stdout: \"\", stderr: \"mkfs: failed to execute mkfs.xfs: No such file or directory\\n\", can't sync mounts: can't create mounts: can't ensure units: unit \"var-lib-persistent\\\\x2dvolumes.mount\" is in a \"failed\" active state]" logger="UnhandledError"
...
I0409 10:57:51.295491       1 record/event.go:389] "Event occurred" object="scylladb-nodepool-1" fieldPath="" kind="NodeConfig" apiVersion="scylla.scylladb.com/v1alpha1" type="Warning" reason="CreateFilesystemFailed" message="Failed to create filesystem xfs on /dev/md/nvmes device: can't run mkfs with args [-t xfs -b size=4096 -K /dev/md127]: exit status 1, stdout: \"\", stderr: \"mkfs: failed to execute mkfs.xfs: No such file or directory\\n\""

What did you expect to happen?

Node setup should be able to create the filesystem.

How can we reproduce it (as minimally and precisely as possible)?

1. Create the GKE cluster with v1.32.3-gke.1170000 cluster version (the current latest, as advised by GKE quickstart in our documentation).
2. Follow the GitOps installation guide until the NodeConfig step: https://operator.docs.scylladb.com/master/installation/gitops.html.
3. Verify that the NodeConfig is in a degraded state.

Scylla Operator version

master

Kubernetes platform name and version

```console $ kubectl version Client Version: v1.32.3 Kustomize Version: v5.5.0 Server Version: v1.32.3-gke.1170000 ```

Kubernetes platform info:
GKE with v1.32.3-gke.1170000 cluster-version.

Please attach the must-gather archive.

must-gather.tar.gz

Anything else we need to know?

No response

[Erouan50]

Hello there!

We're facing the same issue (we use 1.32.2-gke.1182001). I quickly troubleshoot, and from what I gathered, Google removed mkfs from the Container OS Ubuntu image (the node setup daemon set assumes mkfs is installed on the node OS).

I noticed in the source code this nodeSetupImage in the default config with a "node setup" image as value. I checked and found mkfs installed in the image. Are you planning on merging the operator image with this image?

Edit: Never mind, with the chroot in the command, mkfs won't be present anyway even if the images are merged or mkfs installed.