Installer succeeds on RHEL/Rocky 8 but binary fails at runtime: GLIBC_2.32/2.33/2.34 not found

[gdziuba]

Summary

KubeSolo v1.1.6 installer passes all pre-flight checks on Rocky Linux 8.10 (RHEL 8 family), installs the binary, starts the systemd service — and then immediately crash-loops because the binary requires GLIBC ≥ 2.32/2.33/2.34 while Rocky/RHEL 8 ships GLIBC 2.28.

Environment

• OS: Rocky Linux 8.10 (Green Obsidian) — RHEL 8 family (platform:el8)
• Kernel: 4.18.0-553.97.1.el8_10.x86_64
• GLIBC version: 2.28 (shipped with el8)
• KubeSolo version: 1.1.6

Installer output (passes all checks)

```
🔍 Detected glibc system
🔍 Detected init system: systemd
🔍 Detected environment: standard
🔍 Checking for Docker prerequisite conflicts...
✅ No Docker installation detected
🔍 Checking hostname RFC 1123 compliance...
✅ Hostname 'internal-edgehub-demo-aio' is RFC 1123 compliant
🔍 Checking iptables xt_comment module support...
✅ iptables with xt_comment module support verified
🔍 Checking for required cgroups controllers...
✅ All required cgroups v2 controllers are available: cpuset cpu io memory hugetlb pids rdma
🔍 Checking for running KubeSolo processes...
✅ No running KubeSolo processes found
...
✅ kubesolo installation completed!
```

Runtime failure (journalctl -u kubesolo)

```
kubesolo[8585]: /usr/local/bin/kubesolo: /lib64/libc.so.6: version GLIBC_2.33' not found (required by /usr/local/bin/kubesolo) kubesolo[8585]: /usr/local/bin/kubesolo: /lib64/libc.so.6: version GLIBC_2.32' not found (required by /usr/local/bin/kubesolo)
kubesolo[8585]: /usr/local/bin/kubesolo: /lib64/libc.so.6: version GLIBC_2.34' not found (required by /usr/local/bin/kubesolo) systemd[1]: kubesolo.service: Failed with result 'exit-code'. \``

Service enters a crash-restart loop indefinitely.

Root cause

The installer correctly detects a glibc-based system but does not check the glibc version. RHEL/Rocky 8 ships GLIBC 2.28; KubeSolo 1.1.6 requires ≥ 2.34 (first shipped in RHEL/Rocky 9, Ubuntu 22.04+, Debian 12+).

Note: cgroups v2 was a separate prerequisite also not met by default on el8 (requires systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all kernel parameter + reboot). After fixing cgroups, the installer passes all checks but the binary still fails.

Suggested fix

Add a glibc version check to the installer pre-flight, e.g.:

```bash
GLIBC_VERSION=$(ldd --version | head -1 | grep -oE '[0-9]+.[0-9]+$')
REQUIRED="2.32"
if ! awk "BEGIN{exit !($GLIBC_VERSION >= $REQUIRED)}"; then
echo "❌ Error: GLIBC $GLIBC_VERSION found, but KubeSolo requires >= $REQUIRED"
echo " RHEL/Rocky 8 ships GLIBC 2.28. Please use RHEL/Rocky 9, Ubuntu 22.04+, or Debian 12+."
exit 1
fi
```

This prevents the confusing "install succeeds, service immediately crash-loops" experience and gives operators an actionable error message.

Impact

Any operator attempting to install KubeSolo on RHEL 8 / Rocky 8 / CentOS Stream 8 will hit this silently passing install + crash-loop pattern. These are common enterprise Linux distributions still in wide use.

[gdziuba]

Additional finding: musl amd64 tarball contains the glibc binary (packaging bug)

After filing this issue I attempted to work around the glibc version requirement by manually installing kubesolo-v1.1.6-linux-amd64-musl.tar.gz instead of the standard amd64 tarball.

Result: same GLIBC_2.32/2.33/2.34 errors.

Verification:

$ file /usr/local/bin/kubesolo        # extracted from musl tarball
dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2   ← glibc loader, not musl

$ file /usr/local/bin/kubesolo.glibc.bak  # original from installer
dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2

$ # Compare BuildIDs:
BuildID[sha1]=64b7c66b776838a3f944fa4f42150d7336f86037   # musl tarball
BuildID[sha1]=64b7c66b776838a3f944fa4f42150d7336f86037   # glibc tarball

Identical binaries. The kubesolo-v1.1.6-linux-amd64-musl.tar.gz release asset appears to contain the same glibc-linked binary as kubesolo-v1.1.6-linux-amd64.tar.gz. A true musl build would use interpreter /lib/ld-musl-x86_64.so.1 and would not reference GLIBC versioned symbols.

This means there is currently no path to run KubeSolo v1.1.6 on RHEL/Rocky 8 from the published release assets, even with manual binary substitution.

[stevensbkang]

@gdziuba thanks for the issue with details, will get back to you on this after doing some testing!

[atakanatamert]

Hey, I can pick this up if no one's on it yet.

Reproduced on a clean rockylinux:8 container:

$ docker run --rm rockylinux:8 ldd --version | head -1

-> ldd (GNU libc) 2.28

$ docker run --rm rockylinux:8 bash -c 'curl -sfL https://github.com/portainer/kubesolo/releases/download/v1.1.6/kubesolo-v1.1.6-linux-amd64.tar.gz | tar xz -C /tmp && /tmp/kubesolo --version'

-> /tmp/kubesolo: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/kubesolo)
-> /tmp/kubesolo: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /tmp/kubesolo)
-> /tmp/kubesolo: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/kubesolo)

One small addition to the original report: the floor seems to be 2.34. Pulled it straight off the binary's symbol table to be sure:

$ objdump -T /tmp/kubesolo | grep -oP 'GLIBC_\K[0-9.]+' | sort -Vu | tail -1

-> 2.34

So the cutoff lands exactly at RHEL/Rocky 9, Ubuntu 22.04, and Debian 12.

Looking at install.sh, the libc detection only branches musl vs glibc (the ld-musl-*.so.1 check in download_bundle) and never checks the glibc version on the glibc path. So el8 gets a binary it can't run, the install reports success, then systemd crash-loops it.

Two ways to go and I'd rather not guess which you'd prefer:

1. Installer guard: Add a check_glibc_version() next to the existing checks (check_cgroups, check_docker_prerequisite, etc.) that aborts with a clear message when glibc < 2.34, rather than installing a binary the system can't actually run.

2. Actually support el8 by building the glibc artifacts against an older baseline (2.28 based build image, or something like zig cross compile to target an older glibc). This option though may affect the release pipeline.

Let me know which one works for you and I'll pick it up.

[atakanatamert]

Just spotted #139 migrating the installer to Go looks like install.sh is going away, so a bash check here wouldn't be needed. From the PR description the new preflight package (internal/cli/preflight/) has root/hostname/docker/iptables/cgroups/ports but no glibc version check, so this issue would still slip through on el8 after the migration. Happy to add a glibc-version preflight check there once #139 lands (or against that branch if you'd prefer). Same 2.34 floor, just in Go instead of bash.