up

Author: mitchross

CLAUDE.md

@@ -515,6 +515,45 @@ spec:
 
 **Reference**: `infrastructure/storage/csi-driver-nfs/storage-class.yaml` (immich static PV)
 
+### NFS 10G Performance Tuning (CRITICAL)
+
+The Linux kernel (5.4+) defaults NFS `read_ahead_kb` to **128 KB**, which limits sequential NFS reads to ~140 MB/s regardless of link speed. This is because the VFS readahead window only allows ~1 NFS READ (1MB rsize) in flight at a time.
+
+**Fix applied in Talos machine config** (`omni/cluster-template/cluster-template.yaml`):
+
+| Setting | Purpose | Where |
+|---------|---------|-------|
+| `udev rule: ATTR{read_ahead_kb}="16384"` | Sets NFS readahead to 16MB on mount | `machine.udev.rules` (cluster patch) |
+| `siderolabs/nfsrahead` extension | Kernel nfsrahead tool + udev rule | `systemExtensions` (all node types) |
+| `sunrpc.tcp_slot_table_entries: "128"` | Max outstanding RPCs per connection | `machine.sysctls` (cluster patch) |
+| `net.ipv4.tcp_congestion_control: bbr` | Better congestion algorithm for 10G | `machine.sysctls` (cluster patch) |
+| NIC ring buffers = 8192 | Max ring buffer on Proxmox + TrueNAS | Applied on both hosts (persisted) |
+
+**NFS mount options** (set per-PV via CSI `mountOptions`):
+- `nconnect=16` — 16 TCP connections per mount
+- `rsize=1048576` / `wsize=1048576` — 1MB per NFS READ/WRITE op
+- `nfsvers=4.1` — NFSv4.1 with session slots
+- `noatime` — skip access time updates
+
+**Debugging NFS performance**:
+```bash
+# Check readahead (should be 16384, NOT 128)
+kubectl exec -n <ns> <pod> -- cat /sys/class/bdi/0:*/read_ahead_kb
+
+# Check sunrpc slot table (should be 128, NOT 2)
+kubectl exec -n <ns> <pod> -- cat /proc/sys/sunrpc/tcp_slot_table_entries
+
+# Check mount options (verify nconnect=16, rsize=1048576)
+kubectl exec -n <ns> <pod> -- cat /proc/self/mountstats | grep -A3 "192.168.10.133"
+
+# Full NFS stats (connection distribution, slot usage, RTT)
+kubectl exec -n <ns> <pod> -- cat /proc/self/mountstats
+
+# Server-side debugging
+scripts/debug-nfs-server.sh   # Run on TrueNAS SSH
+scripts/debug-nfs-client.sh   # Run on Proxmox SSH
+```
+
 ## Automated Backup & Restore with Kyverno
 
 ### The Magic Label Pattern

docs/network-topology.md

@@ -115,7 +115,49 @@ showmount -e 192.168.10.133
 ### Storage Performance Testing
 
 ```bash
-# Test 10G link to TrueNAS
-kubectl exec -n <ns> <pod> -- dd if=/dev/zero of=/mnt/nfs/test bs=1G count=1
-# Should see ~1GB/s+ throughput on 10G link
+# Test raw wire speed (should be ~9.4 Gbps)
+iperf3 -c 192.168.10.133
+
+# Test NFS throughput from inside a pod
+kubectl exec -n <ns> <pod> -- dd if=/mnt/nfs/testfile of=/dev/null bs=1M status=progress
+
+# Test NFS throughput from Proxmox host (bypasses VM layer)
+mount -t nfs -o nfsvers=4.1,nconnect=16,rsize=1048576,wsize=1048576 192.168.10.133:/mnt/BigTank/k8s/llama-cpp /mnt/nfstest
+dd if=/mnt/nfstest/testfile of=/dev/null bs=1M status=progress
 ```
+
+### NFS 10G Tuning
+
+The default Linux kernel `read_ahead_kb` of 128 KB limits NFS sequential reads to ~140 MB/s on any link speed. The cluster applies these fixes via Talos machine config:
+
+| Layer | Setting | Value |
+|-------|---------|-------|
+| **VFS readahead** | udev rule `ATTR{read_ahead_kb}` | 16384 (16MB) |
+| **NFS readahead** | `siderolabs/nfsrahead` extension | Installed on all nodes |
+| **RPC concurrency** | `sunrpc.tcp_slot_table_entries` | 128 (default was 2) |
+| **TCP congestion** | `net.ipv4.tcp_congestion_control` | bbr |
+| **TCP buffers** | `net.core.rmem_max` / `wmem_max` | 64MB |
+| **NIC ring buffers** | Proxmox + TrueNAS | 8192 (max) |
+| **NFS mount options** | Per-PV CSI mountOptions | `nconnect=16,rsize=1M,wsize=1M` |
+
+**Verified performance** (from TrueNAS ARC-cached 4GB file):
+
+| Layer | Speed |
+|-------|-------|
+| iperf3 (wire) | 9.4 Gb/s |
+| Proxmox host → NFS | 2.7 GB/s |
+| Talos VM → NFS (before tuning) | ~128 MB/s |
+
+**Debug commands**:
+```bash
+# Verify readahead is 16384 (not 128)
+kubectl exec -n <ns> <pod> -- cat /sys/class/bdi/0:*/read_ahead_kb
+
+# Verify sunrpc slots are 128 (not 2)
+kubectl exec -n <ns> <pod> -- cat /proc/sys/sunrpc/tcp_slot_table_entries
+
+# Full NFS mount stats (connections, slots, RTT)
+kubectl exec -n <ns> <pod> -- cat /proc/self/mountstats
+```
+
+See `scripts/debug-nfs-server.sh` (TrueNAS) and `scripts/debug-nfs-client.sh` (Proxmox) for comprehensive debugging.

my-apps/ai/README.md

@@ -277,7 +277,7 @@ All routes use `gateway-internal` (Cilium Gateway API). LLM and Open WebUI route
 | ComfyUI | NFS (static PV, CSI) | 250Gi | `192.168.10.133:/mnt/BigTank/k8s/comfyui` |
 | Open WebUI | Longhorn | 5Gi | Dynamic PVC |
 
-NFS mounts use `nconnect=16` over 10G for fast model loading.
+NFS mounts use `nconnect=16` over 10G for fast model loading. Performance depends on Talos kernel tuning — `read_ahead_kb` must be 16384+ (set via udev rule in cluster template) and `sunrpc.tcp_slot_table_entries` must be 128+ (set via sysctl). Without these, NFS caps at ~140 MB/s regardless of link speed.
 
 ## Caveats
 

omni/cluster-template/cluster-template.yaml

@@ -119,7 +119,6 @@ machineClass:
 systemExtensions:
 - siderolabs/iscsi-tools
 - siderolabs/nfs-utils
-- siderolabs/nfsrahead
 - siderolabs/qemu-guest-agent
 - siderolabs/util-linux-tools
 - siderolabs/nonfree-kmod-nvidia-production