Skip to content

Optimize exporter CPU and memory usage for high-density nodes - #351

Merged
Lyt99 merged 7 commits into
alibaba:mainfrom
jzwlqx:optimize
Apr 9, 2026
Merged

Optimize exporter CPU and memory usage for high-density nodes#351
Lyt99 merged 7 commits into
alibaba:mainfrom
jzwlqx:optimize

Conversation

@jzwlqx

@jzwlqx jzwlqx commented Apr 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

This MR addresses high CPU (~367m) and memory (~300Mi) consumption of kubeskoop-agent on nodes
with many pods (60+). Through a series of targeted optimizations, CPU is reduced by 72% to ~101m
and memory by 58% to ~126Mi.

Changes

1. LinkManager abstraction (linkmanager.go)

  • Centralized network interface management with netlink subscription for link events
  • Replaced per-probe ad-hoc interface discovery in flow and nlqdisc probes
  • Eliminated redundant netlink connections and polling

2. Switch from bswang/netlink fork to upstream vishvananda/netlink

  • The forked netlink library had a busy-loop bug (nonblocking socket without epoll registration)
    causing unnecessary CPU consumption
  • Migrated to the official upstream library

3. Optimize proctcpsummary /proc/net/tcp parsing (proctcp.go)

  • Replaced regexp-based parsing with single-pass zero-allocation field extraction
  • Added 2-second TTL cache to avoid redundant collection across metrics

4. NETLINK_SOCK_DIAG for TCP socket collection (sockdiag.go)

  • Implemented binary SOCK_DIAG protocol (inet_diag) as replacement for text-based /proc/net/tcp
  • 2.3x faster per-collection (713ms -> 306ms avg), 2.34x less kernel CPU
  • Automatic fallback to /proc parsing if SOCK_DIAG is unavailable
  • Benchmark at scale: up to 5x speedup at 10K+ sockets (p50)

5. Lazy-load kernel symbols (kallsyms.go)

  • Changed unconditional init() loading of 214K symbols (~30MB) to sync.Once lazy loading
  • Symbols only loaded when BPF probes (tracepacketloss, tracetcpretrans) are active
  • Saves ~30MB heap for the common case where no BPF probes are enabled

6. Eliminate intermediate string map allocations (procnetstat.go, procsnmp.go)

  • procnetstat: replaced map[string]map[string]string -> direct uint64 parsing, only extracting
    needed fields. Reduced cumulative allocations from ~7.7GB to near zero.
  • procsnmp: same optimization for /proc/net/snmp (tcp/udp/ip protocols). Reduced ~550MB
    cumulative allocations.

Performance Results (node with 63 pods, 59 unique namespaces)

Metric Before After Reduction
Pod CPU 367m 101m -72%
Pod Memory (RSS) ~300Mi 126Mi -58%
Go Heap (live) 52.7MB 32MB -39%
GC cycles (2min) 1431 304 -79%
TCP collection (p50) 687ms 210ms -70%
Kernel CPU per coll. 38.3s 16.4s -57%

Test Plan

  • Unit tests for SOCK_DIAG parsing (sockdiag_test.go) and proc parsing (proctcp_test.go)
  • All 10 configured probes verified producing correct metrics post-deploy
    (conntrack, qdisc, netdev, io, sock, tcpsummary, tcp, tcpext, udp, rdma)
  • Metrics cross-validated between problem node (.51, 63 pods) and normal nodes
  • pprof heap/cpu profiling before and after each optimization
  • Standalone benchmark tool (cmd/tcpbench) validated SOCK_DIAG vs proc at
    0/1K/10K/50K socket scales
  • No new errors in inspector logs (rdma errors are pre-existing, unrelated)

@jzwlqx
jzwlqx requested review from BSWANG and Lyt99 April 7, 2026 08:23
@jzwlqx jzwlqx added the enhancement New feature or request label Apr 8, 2026
@Lyt99
Lyt99 merged commit a75c0f6 into alibaba:main Apr 9, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants