Skip to content

feat(ipvs): add IPVS load balancer monitoring support#325

Open
RichardoMrMu wants to merge 1 commit into
hengyoush:mainfrom
RichardoMrMu:feature/ipvs-monitoring
Open

feat(ipvs): add IPVS load balancer monitoring support#325
RichardoMrMu wants to merge 1 commit into
hengyoush:mainfrom
RichardoMrMu:feature/ipvs-monitoring

Conversation

@RichardoMrMu

Copy link
Copy Markdown
  • Add eBPF kprobe/kretprobe for IPVS kernel functions
  • Track IPVS call chain: CONN_NEW -> SCHEDULE -> NAT_XMIT -> CONN_PUT
  • Display VIP to RealServer mapping in flow chart
  • Support NAT/DR/TUNNEL forwarding modes
  • Add --enable-ipvs flag to enable IPVS tracing

New files:

  • agent/ipvs/: IPVS tracker, cache, and event handling
  • bpf/ipvs.*: eBPF programs for IPVS function tracing

Modified files:

  • agent/agent.go: Initialize IPVS tracker
  • agent/common/options.go: Add EnableIPVS option
  • agent/analysis/common/types.go: Add IPVS fields to AnnotatedRecord
  • agent/analysis/stat.go: Query IPVS cache when creating records
  • agent/render/watch/time_detail.go: Display IPVS info in flow chart
  • cmd/watch.go: Add --enable-ipvs flag
image

- Add eBPF kprobe/kretprobe for IPVS kernel functions
- Track IPVS call chain: CONN_NEW -> SCHEDULE -> NAT_XMIT -> CONN_PUT
- Display VIP to RealServer mapping in flow chart
- Support NAT/DR/TUNNEL forwarding modes
- Add --enable-ipvs flag to enable IPVS tracing

New files:
- agent/ipvs/: IPVS tracker, cache, and event handling
- bpf/ipvs.*: eBPF programs for IPVS function tracing

Modified files:
- agent/agent.go: Initialize IPVS tracker
- agent/common/options.go: Add EnableIPVS option
- agent/analysis/common/types.go: Add IPVS fields to AnnotatedRecord
- agent/analysis/stat.go: Query IPVS cache when creating records
- agent/render/watch/time_detail.go: Display IPVS info in flow chart
- cmd/watch.go: Add --enable-ipvs flag
@vercel

vercel Bot commented Feb 9, 2026

Copy link
Copy Markdown

Someone is attempting to deploy a commit to the hengyoush's projects Team on Vercel.

A member of the Team first needs to authorize it.

@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. enhancement New feature or request labels Feb 9, 2026
@RichardoMrMu

Copy link
Copy Markdown
Author

Summary

This PR adds IPVS (IP Virtual Server) load balancer monitoring support to kyanos. It enables users to trace IPVS kernel function calls and visualize the VIP to RealServer mapping in the network flow chart.

Background

Problem

In Kubernetes environments using IPVS mode for kube-proxy, network traffic goes through IPVS load balancing before reaching the actual backend pods. Currently, kyanos cannot show this IPVS layer in the network flow visualization, making it difficult to:

  1. Understand the complete network path from client to backend
  2. Identify IPVS-related latency issues
  3. Debug load balancing problems

Solution

Add eBPF-based IPVS tracing that:

  • Hooks into IPVS kernel functions using kprobe/kretprobe
  • Tracks the complete IPVS call chain with latency for each function
  • Associates IPVS information with network connections
  • Displays IPVS load balancing info in the flow chart

Features

1. IPVS Function Tracing

Traces 8 key IPVS kernel functions:

Function Event Type Description
ip_vs_conn_new CONN_NEW New IPVS connection
ip_vs_conn_in_get CONN_IN Inbound connection lookup
ip_vs_conn_out_get CONN_OUT Outbound connection lookup
ip_vs_schedule SCHEDULE Backend server selection
ip_vs_nat_xmit NAT_XMIT NAT mode forwarding
ip_vs_dr_xmit DR_XMIT Direct Routing mode
ip_vs_tunnel_xmit TUNNEL_XMIT Tunnel mode forwarding
ip_vs_conn_put CONN_PUT Connection reference release

2. Flow Chart Integration

IPVS information is displayed in the flow chart:

[IPVS LB] 10.96.49.40:9000 -> 172.24.0.39:9000 [NAT] (0.03ms)
[IPVS Call Chain] CONN_NEW(5.2us) -> SCHEDULE(8.3us) -> NAT_XMIT(12.1us) -> CONN_PUT(4.5us)

+-----------------------+   +-------------------+   +-------------------------+
| Process(pid:1681358)  |   | IPVS[NAT](0.03ms) |   | eth0@if2716(used:0.00ms)|
|                       --->|                   --->|                         |
+-----------------------+   +-------------------+   +-------------------------+

3. Usage

# Enable IPVS monitoring
./kyanos watch --remote-ports 9000 --enable-ipvs

Technical Design

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         User Space                               │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐         │
│  │   IPVS      │    │   IPVS      │    │   IPVS      │         │
│  │   Tracker   │───▶│   Cache     │───▶│   Display   │         │
│  └─────────────┘    └─────────────┘    └─────────────┘         │
│         ▲                  │                  │                 │
│         │                  ▼                  ▼                 │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐         │
│  │   Perf      │    │ Annotated   │    │   TUI       │         │
│  │   Reader    │    │   Record    │    │   Render    │         │
│  └─────────────┘    └─────────────┘    └─────────────┘         │
├─────────────────────────────────────────────────────────────────┤
│                         Kernel Space                             │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────────────────────────────────────────────────┐   │
│  │                    eBPF Programs                         │   │
│  │  kprobe/kretprobe: ip_vs_conn_new, ip_vs_schedule, ...  │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Key Components

  1. eBPF Programs (bpf/ipvs.bpf.c, bpf/ipvs.h)

    • kprobe/kretprobe for IPVS kernel functions
    • Extract connection info (VIP, RealServer, protocol, flags)
    • Calculate function latency
  2. IPVS Tracker (agent/ipvs/tracker.go)

    • Load and attach eBPF programs
    • Read events from perf buffer
    • Build call chains from events
  3. IPVS Cache (agent/ipvs/cache.go)

    • Store completed IPVS call chains
    • Lookup by VIP or RealServer
  4. Integration (agent/analysis/stat.go, agent/render/watch/time_detail.go)

    • Associate IPVS info with AnnotatedRecord
    • Display in flow chart

Changes

New Files

File Description
agent/ipvs/ipvs.go IPVS module entry point
agent/ipvs/cache.go Global cache for IPVS chains
agent/ipvs/event.go Event types and chain structures
agent/ipvs/tracker.go eBPF program management
bpf/ipvs.bpf.c eBPF C source code
bpf/ipvs.h eBPF header file
bpf/ipvs_events.go Event parsing helpers
bpf/ipvs_arm64_bpfel.go Generated eBPF code (ARM64)
bpf/ipvs_x86_bpfel.go Generated eBPF code (x86_64)

Modified Files

File Changes
agent/agent.go Initialize IPVS tracker
agent/common/options.go Add EnableIPVS option
agent/analysis/common/types.go Add IPVS fields to AnnotatedRecord
agent/analysis/stat.go Query IPVS cache when creating records
agent/render/watch/time_detail.go Display IPVS info in flow chart
bpf/gen.go Add IPVS BPF generation
cmd/watch.go Add --enable-ipvs flag

Testing

Environment

  • Linux kernel >= 4.15 (eBPF support)
  • IPVS module loaded (modprobe ip_vs)
  • Kubernetes cluster with IPVS mode kube-proxy

Test Steps

  1. Check IPVS module: lsmod | grep ip_vs
  2. Check IPVS rules: ipvsadm -Ln
  3. Run kyanos: ./kyanos watch --remote-ports 9000 --enable-ipvs
  4. Generate traffic to a Service VIP
  5. Verify IPVS info appears in flow chart

Test Results

  • ✅ IPVS probes attached successfully (16/16)
  • ✅ IPVS events captured with correct IP/port
  • ✅ Call chains built and cached
  • ✅ IPVS info displayed in flow chart

@RichardoMrMu

Copy link
Copy Markdown
Author

@hengyoush hi, can u review this code?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant