Skip to content

v0.9.0: monitoring servers bound to localhost breaks Kubernetes liveness probes #458

Description

@gbhall

Summary

v0.9.0 binds monitoring servers (Prometheus metrics on port 9273, profiler on port 4269) to localhost. This breaks Kubernetes liveness/readiness probes, which connect via the pod IP, not localhost.

Environment

  • Kubernetes: GKE 1.34.4
  • Deployment: Helm chart with liveness probe on :9273/metrics
  • v0.8.0: stable, no restarts
  • v0.9.0: exits with code 2 every ~3 minutes (liveness probe failure)

Steps to Reproduce

  1. Deploy v0.9.0 on Kubernetes with the default Helm chart
  2. The Helm chart configures a liveness probe: httpGet :9273/metrics
  3. The kubelet sends the probe from the node to the pod IP (e.g. 10.48.1.x:9273)
  4. Since v0.9.0 binds the metrics server to 127.0.0.1, the connection is refused
  5. After failureThreshold consecutive failures, Kubernetes kills the container

Evidence

Pod events show continuous probe failures:

Warning  Unhealthy  Liveness probe failed: Get "http://10.48.1.x:9273/metrics": dial tcp 10.48.1.x:9273: connect: connection refused
Normal   Killing    Container fleet-telemetry failed liveness probe, will be restarted

Pod uses minimal resources (9m CPU, 11Mi memory) — not a resource issue.
Downgrading to v0.8.0 immediately resolves the issue (0 restarts).

Suggested Fix

Bind the monitoring servers to 0.0.0.0 (or make the bind address configurable), while keeping the security improvements from the release. Alternatively, update the Helm chart's liveness probe to use host: 127.0.0.1 if localhost binding is intentional.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions