Skip to content

soft serve doesn't reap child git processes, causing zombie accumulation #891

@eznix86

Description

@eznix86

Description:

soft serve spawns git child processes to handle operations but doesn't reap them, causing zombie processes to accumulate until the host runs out of PIDs or fork/exec fails.

Reproduction:

  1. Run soft serve on any machine (container or bare metal)
  2. Perform git operations against it (clone, push, pull via SSH or git protocol)
  3. Observe zombie processes accumulating under the soft parent:
# ps aux | awk '$8 ~ /Z/'
root  2592996  ... [git] <defunct>
root  2593008  ... [git] <defunct>
root  2593023  ... [git] <defunct>
...

After enough operations, PID exhaustion occurs:
fork/exec /proc/self/fd/6: resource temporarily unavailable

Root cause:

The Go process does not appear to handle SIGCHLD or call waitpid()/wait4() on its child processes. Sending SIGCHLD to the soft process manually reaps all zombies immediately.
kill -s SIGCHLD $(pgrep soft)

Impact:

  • In Kubernetes, this made nodes unusable within hours (30k+ zombies in under 24h)
  • All pods on the affected node fail with CreateContainerError: resource temporarily unavailable
  • The only recovery was a full node reboot

Expected behavior:

Child git processes should be reaped automatically — either via a SIGCHLD handler with waitpid(-1, ...) in a loop, or by running the process with an init supervisor like tini or catatonio that handles reaping.

Environment:

  • charmcli/soft-serve:latest (image sha256:7742f301ab4e5c24eccf999b10aabb97ec64e18c23f693a16599071d49ae5bf4)
  • Running inside a Kubernetes container (containerd 1.7.30)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions