[argo-cd] dex-server CrashLoopBackOff (SIGSEGV exit 139) — chart-default dex tag (alpine/musl) is incompatible with chart-default argocd init image (glibc)

[danielfree19]

Summary

argocd-dex-server lands in CrashLoopBackOff with exit code 139 (SIGSEGV) and no log output, because the argocd binary that the init container copies into the dex container is glibc-linked, while the upstream ghcr.io/dexidp/dex:vX.Y.Z image is built on Alpine (musl libc). The musl loader cannot load the glibc binary, so the kernel kills the process at exec time before argocd-dex ever reaches main().

This is silently dependent on dex's base-image choice — a unilateral upstream change in dex (e.g. an alpine version bump that re-pushes the same :vX.Y.Z tag) is enough to break every cluster that pulls a fresh image.

Repro

Running argo-cd Helm chart 9.4.1 (default dex.image.tag: v2.44.0) with quay.io/argoproj/argocd:v3.3.0, no image overrides for dex:

dex:
  enabled: true
  # leave image unpinned -> chart default vX.Y.Z (alpine)

Pod status:

argocd-dex-server-...  0/1  CrashLoopBackOff  N (Nmin ago)  ...
Last State: Terminated, Reason: Error, Exit Code: 139

No stdout/stderr from the container. Same shape on chart 9.5.12 (default dex.image.tag: v2.45.1, also alpine).

This is a chart-default failure, not a user misconfiguration

Both ends of the libc contract come from the chart's own templates/dex/deployment.yaml (chart 9.5.12):

• Init container (copyutil, ~line 178): image defaults to global.image.repository:argo-cd.defaultTag — i.e. the argocd image (Ubuntu/glibc), running /bin/cp -n /usr/local/bin/argocd /shared/argocd-dex.
• Main dex container (~line 78): runs /shared/argocd-dex rundex from dex.image.repository:dex.image.tag — defaults to ghcr.io/dexidp/dex:v2.45.1 on chart 9.5.12 (Alpine/musl).

A fresh helm install with no overrides produces the broken combo. The chart picks both endpoints but does not assert that they share an ABI.

Diagnostic evidence

In a debug pod replicating the dex deployment's init+main containers, exec'ing into the dex container:

/ # ldd /shared/argocd-dex
/lib/ld-musl-x86_64.so.1: /shared/argocd-dex: Not a valid dynamic program

/ # cat /etc/os-release
PRETTY_NAME="Alpine Linux v3.22"

/ # ls /lib/ld-musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2
/lib/ld-musl-x86_64.so.1
ls: /lib64/ld-linux-x86-64.so.2: No such file or directory

For comparison, in the argocd image (quay.io/argoproj/argocd:v3.3.0):

PRETTY_NAME="Ubuntu 25.04"
/lib64/ld-linux-x86-64.so.2

Confirmed identical for quay.io/argoproj/argocd:v3.4.1 (Ubuntu 25.10, glibc).

The argocd binary appears static to glibc's ldd (not a dynamic executable) but in practice fails to load on a musl-only filesystem — likely Go's runtime / CGO net resolver picks up glibc-isms that musl's loader rejects.

Reproduction with the distroless dex variant works:

dex:
  image:
    tag: v2.44.0-distroless    # gcr.io/distroless/static-debian12, glibc

→ pod starts, dex listens on :5556, OIDC works.

Why this is silent and recurring

• :vX.Y.Z tags on ghcr.io/dexidp/dex are mutable. Within v2.44.0's release cycle the maintainers bumped alpine 3.21.3 → 3.22.0 → 3.22.1 and re-pushed the same tag. So a cluster that's been running fine on a cached digest can break the next time a node has to pull (reschedule, image-cache eviction, new node).
• Nothing in the argocd or chart docs flags that the dex.image.tag must be the distroless variant.
• The init container produces no error (cp succeeds); the dex container produces no error (kernel kills the process before any code runs). The only signal is exit 139.

Suggested fixes

1. Pin chart default to -distroless in argo-helm/charts/argo-cd (dex.image.tag: vX.Y.Z-distroless). Smallest, most direct fix.
2. Pin by digest in chart defaults so an upstream re-tag can't break consumers between releases.
3. Document the libc compatibility requirement of the init-container copy pattern in the dex section of the chart README and in the argocd OIDC docs.
4. Long-term: ship a static (CGO-disabled) argocd-dex subcommand binary — or split it into a tiny dedicated image — so the pattern works regardless of dex's base-image choice. The "share a binary across containers via a volume" pattern is the real footgun.

Versions

• argocd image: quay.io/argoproj/argocd:v3.3.0 (Ubuntu 25.04, glibc) — observed crashing
• argocd chart deployed: argo-cd 9.4.1
• argocd chart staged for upgrade: argo-cd 9.5.12 (default dex.image.tag: v2.45.1 — same alpine/musl base, same crash shape)
• dex image deployed: ghcr.io/dexidp/dex:v2.44.0 (Alpine 3.22, musl) — digest sha256:587f29be7f1437c7123ecb9ff9e31079ded140365c3518a8e85b99fae5f22421
• Latest released argocd v3.3.x at time of report: v3.3.9 (2026-04-30) — also Ubuntu/glibc, same problem
• Kubernetes: OKE on Oracle Linux 8.10 nodes, amd64