Skip to content

Commit aad0c01

Browse files
authored
fix(containerd): skip dangling-pod check when kubelet read-only port … (#1241)
…is closed The containerd component unconditionally calls ListPodsFromKubeletReadOnlyPort against http://localhost:10255 to cross-check sandboxes against kubelet's pod list, producing a recurring connection-refused error log on every node where the kubelet readOnlyPort is disabled (the kubeadm/CIS-hardened default). Gate the call on netutil.IsPortOpen, matching the pattern the kubelet component already uses (components/kubelet/component.go). When the port is closed the dangling-pod check is skipped (danglingCount stays 0) and the rest of the containerd checks proceed normally. Example of error: ```May 07 11:58:00 cluster-node-2 gpud[3478]: {"level":"error","ts":"2026-05-07T11:58:00.933Z","caller":"log/log.go:183","msg":"error listing pods from kubelet: Get \"http://localhost:10255/pods\": dial tcp [::1]:10255: connect: connection refused","stacktrace":"github.com/leptonai/gpud/pkg/log.(*gpudLogger).Errorf\n\t/home/runner/work/gpud/gpud/pkg/log/log.go:183\ngithub.com/leptonai/gpud/components/containerd.(*component).Check\n\t/home/runner/work/gpud/gpud/components/containerd/component.go:306\ngithub.com/leptonai/gpud/components/containerd.(*component).Start.func1\n\t/home/runner/work/gpud/gpud/components/containerd/component.go:135"} May 07 11:59:00 eden1-2 gpud[3478]: {"level":"error","ts":"2026-05-07T11:59:00.935Z","caller":"log/log.go:183","msg":"error listing pods from kubelet: Get \"http://localhost:10255/pods\": dial tcp [::1]:10255: connect: connection refused","stacktrace":"github.com/leptonai/gpud/pkg/log.(*gpudLogger).Errorf\n\t/home/runner/work/gpud/gpud/pkg/log/log.go:183\ngithub.com/leptonai/gpud/components/containerd.(*component).Check\n\t/home/runner/work/gpud/gpud/components/contai ``` With ``` $ gpud --version gpud version v0.11.4 ``` draft reason: - [ ] And as I'm writing this I haven't tried a binary built with this PR to see that it behaves differently.
1 parent 0451b5f commit aad0c01

1 file changed

Lines changed: 11 additions & 7 deletions

File tree

components/containerd/component.go

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ import (
1818
"github.com/leptonai/gpud/components"
1919
componentkubelet "github.com/leptonai/gpud/components/kubelet"
2020
"github.com/leptonai/gpud/pkg/log"
21+
"github.com/leptonai/gpud/pkg/netutil"
2122
nvidianvml "github.com/leptonai/gpud/pkg/nvidia/nvml"
2223
"github.com/leptonai/gpud/pkg/systemd"
2324
)
@@ -299,13 +300,16 @@ func (c *component) Check() components.CheckResult {
299300
}
300301

301302
var danglingCount int
302-
cctx, ccancel = context.WithTimeout(c.ctx, 30*time.Second)
303-
_, kubeletPods, err := componentkubelet.ListPodsFromKubeletReadOnlyPort(cctx, componentkubelet.DefaultKubeletReadOnlyPort)
304-
ccancel()
305-
if err != nil {
306-
log.Logger.Errorf("error listing pods from kubelet: %v", err)
307-
} else {
308-
danglingCount = danglingPodCount(cr.Pods, kubeletPods)
303+
// skip dangling-pod check if kubelet read-only port is closed
304+
if netutil.IsPortOpen(componentkubelet.DefaultKubeletReadOnlyPort) {
305+
cctx, ccancel = context.WithTimeout(c.ctx, 30*time.Second)
306+
_, kubeletPods, err := componentkubelet.ListPodsFromKubeletReadOnlyPort(cctx, componentkubelet.DefaultKubeletReadOnlyPort)
307+
ccancel()
308+
if err != nil {
309+
log.Logger.Errorf("error listing pods from kubelet: %v", err)
310+
} else {
311+
danglingCount = danglingPodCount(cr.Pods, kubeletPods)
312+
}
309313
}
310314
switch {
311315
case danglingCount > DanglingUnhealthyThreshold:

0 commit comments

Comments
 (0)