Skip to content

Commit 781cdb8

Browse files
authored
feat: Add rolling avg toggle (#2)
* Add rolling avg feature Signed-off-by: Erez Zarum <erezz@amazon.com> * Fix cargo fmt issues, add extra metrics selection Signed-off-by: Erez Zarum <erezz@amazon.com> * Add kubernetes example Signed-off-by: Erez Zarum <erezz@amazon.com> * Implement fixes Signed-off-by: Erez Zarum <erezz@amazon.com> --------- Signed-off-by: Erez Zarum <erezz@amazon.com>
1 parent 077fc65 commit 781cdb8

7 files changed

Lines changed: 854 additions & 86 deletions

File tree

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ communication benchmarks:
4747
- [NVSHMEM](examples/nvshmem/) — one-sided GPU communication
4848
- [PPLX Kernels](examples/pplx/) — MoE all-to-all dispatch/combine
4949
- [RDMA Statistics](examples/rdma/) — shell-based RDMA stats
50+
- [Kubernetes](examples/kubernetes/) — DaemonSet deployment for Kubernetes
5051

5152
## How It Works
5253

examples/kubernetes/Dockerfile

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
FROM rust:1-alpine AS builder
2+
RUN apk add --no-cache musl-dev
3+
WORKDIR /build
4+
COPY Cargo.toml ./
5+
COPY src/ src/
6+
RUN cargo build --release
7+
8+
FROM alpine:3
9+
COPY --from=builder /build/target/release/rdmatop /usr/local/bin/rdmatop
10+
ENTRYPOINT ["/usr/local/bin/rdmatop"]

examples/kubernetes/README.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Kubernetes
2+
3+
Deploy `rdmatop` as a DaemonSet to monitor RDMA traffic across all
4+
GPU nodes in a Kubernetes cluster. The DaemonSet runs a container with
5+
`NET_ADMIN` capability and host PID/network namespaces on each matching
6+
node, giving you per-node RDMA visibility without SSH access.
7+
8+
## Build
9+
10+
Build the container image from the project root:
11+
12+
```bash
13+
cd rdmatop
14+
docker build -f examples/kubernetes/Dockerfile -t rdmatop:latest .
15+
```
16+
17+
Push to your registry if needed:
18+
19+
```bash
20+
docker tag rdmatop:latest <registry>/rdmatop:latest
21+
docker push <registry>/rdmatop:latest
22+
```
23+
24+
## Deploy
25+
26+
Before deploying, update the `image` field in `daemonset.yaml` to point to your registry (e.g. `<registry>/rdmatop:latest`) so Kubernetes can pull the image:
27+
28+
```bash
29+
kubectl apply -f examples/kubernetes/daemonset.yaml
30+
```
31+
32+
The DaemonSet targets GPU instance types (p5, p5e, p5en, p6e, p6-b200,
33+
p6-b300) via node affinity. It uses `hostNetwork: true` so the container
34+
sees the host's RDMA devices, `hostPID: true` so rdmatop can read
35+
`/proc/<pid>` to enrich RDMA QP owners, and requests `NET_ADMIN`
36+
capability for netlink access.
37+
38+
## Usage
39+
40+
Exec into a running pod to launch `rdmatop`:
41+
42+
```bash
43+
# List pods
44+
kubectl get pods -l app=rdmatop
45+
46+
# Attach to a specific node's pod
47+
kubectl exec -it <pod-name> -- rdmatop
48+
```
49+
50+
## Configuration
51+
52+
| Field | Description |
53+
|-------|-------------|
54+
| `hostNetwork: true` | Required — exposes host RDMA devices to the container |
55+
| `hostPID: true` | Required — allows rdmatop to read `/proc/<pid>` for RDMA process info |
56+
| `NET_ADMIN` | Required — allows netlink queries for RDMA stats |
57+
| `nodeAffinity` | Targets GPU instance types; edit `values` to match your cluster |

examples/kubernetes/daemonset.yaml

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
apiVersion: apps/v1
2+
kind: DaemonSet
3+
metadata:
4+
name: rdmatop
5+
labels:
6+
app: rdmatop
7+
spec:
8+
selector:
9+
matchLabels:
10+
app: rdmatop
11+
template:
12+
metadata:
13+
labels:
14+
app: rdmatop
15+
spec:
16+
affinity:
17+
nodeAffinity:
18+
requiredDuringSchedulingIgnoredDuringExecution:
19+
nodeSelectorTerms:
20+
- matchExpressions:
21+
- key: node.kubernetes.io/instance-type
22+
operator: In
23+
values:
24+
- p5.48xlarge
25+
- p5e.48xlarge
26+
- p5en.48xlarge
27+
- p6e-gb200.36xlarge
28+
- p6-b200.48xlarge
29+
- p6-b300.48xlarge
30+
hostNetwork: true
31+
hostPID: true
32+
containers:
33+
- name: rdmatop
34+
image: rdmatop:latest
35+
imagePullPolicy: IfNotPresent
36+
command: ["sleep", "infinity"]
37+
securityContext:
38+
capabilities:
39+
add:
40+
- NET_ADMIN
41+
tolerations:
42+
- operator: Exists

0 commit comments

Comments
 (0)