Ansible Role: net_tuning

Dynamic network performance tuning for Linux servers. Automatically configures RSS/RPS, RFS, XPS, ring buffers, NIC queues, conntrack, and sysctl based on hardware detection.

What it does

All parameters are computed dynamically from CPU count, NIC speed, queue count, and driver capabilities:

Feature	Formula	Effect
RSS/RPS	Smart mode selection (see below)	Distributes RX packet processing
RFS	CPU * flows_per_cpu / num_queues	Steers packets to the CPU running the application
XPS	groups (few queues) or 1:1 (many queues)	Distributes TX across CPUs
NIC queues	min(max_hw, nproc, 32)	Optimizes hardware RSS queues
Ring buffers	min(128 * nproc, max_hw)	Reduces packet drops during bursts
conntrack	max(524288, nproc * 32768)	Dynamic connection tracking table
netdev_budget	clamp(nproc * 64, 300, 4096)	Scales softirq processing capacity
busy_poll	only supported drivers	Reduces socket read latency
sysctl	tcp_fastopen, tcp_tw_reuse, etc.	TCP stack tuning

Smart RSS/RPS mode selection

The role automatically chooses the optimal packet steering mode based on hardware:

if HW_queues >= 16:
    → RSS mode (hardware distribution sufficient)
elif max_pps_per_queue > 200K (10Gbps+ NICs):
    → RPS mode (CPU bottleneck risk)
elif CPU/queue ratio > 8:1:
    → RPS mode (significant IRQ imbalance)
else:
    → RSS mode (low PPS, RPS overhead not justified)

This avoids unnecessary IPI overhead on servers where RSS handles distribution well enough (e.g., 1Gbps NICs with adequate queue counts), while enabling RPS where it's truly needed (few queues on many-CPU servers, or 10Gbps+ NICs).

Interface detection

Only configures physical PCI NICs that are UP:

• Verifies PCI bus via ethtool -i (bus-info starts with 0000:)
• Skips: VLANs, USB-ethernet (BMC/IPMI), virtual interfaces, DOWN ports
• ntuple disabled to prevent CFA flow table exhaustion on Broadcom NICs
• busy_poll enabled only for drivers that support NAPI busy polling

Conntrack tuning

Dynamic connection tracking table sizing prevents nf_conntrack: table full under load:

• nf_conntrack_max = max(524288, nproc * 32768)
• hashsize = nf_conntrack_max / 4
• Aggressive timeouts for highload (established=600s, time_wait=30s, close_wait=10s)
• Applied before NIC changes to prevent overflow during interface reset

Supported NIC drivers

Driver	NIC	busy_poll	Multi-queue
tg3	Broadcom BCM5720	no	no
r8169	Realtek 8168	no	no
igb	Intel I210/I211	yes	yes (2-8 queues)
bnxt_en	Broadcom BCM575xx	yes	yes (up to 74 queues)
ixgbe	Intel 10GbE	yes	yes
i40e	Intel XL710	yes	yes
ice	Intel E800	yes	yes
mlx5_core	Mellanox ConnectX	yes	yes
ena	AWS ENA	yes	yes

Requirements

• Linux kernel 3.2+ (RPS/RFS support)
• ethtool installed
• python3 installed (for bitmask calculation)
• Root/sudo access

Role Variables

# RPS/RFS flow table sizing
net_tuning_flows_per_cpu: 4096

# RSS/RPS threshold: HW queues >= this → RSS sufficient
net_tuning_rss_queue_threshold: 16

# NIC queues cap (prevents IRQ storm on high-queue NICs)
net_tuning_max_queues: 32

# Ring buffer sizing
net_tuning_ring_buffer_base: 128

# Conntrack sizing
net_tuning_conntrack_per_cpu: 32768

# Conntrack timeouts (seconds)
net_tuning_conntrack_tcp_established: 600
net_tuning_conntrack_tcp_time_wait: 30
net_tuning_conntrack_tcp_close_wait: 10
net_tuning_conntrack_tcp_fin_wait: 30
net_tuning_conntrack_tcp_close: 10
net_tuning_conntrack_udp: 30
net_tuning_conntrack_udp_stream: 60

# TCP tuning
net_tuning_tcp_slow_start_after_idle: 0
net_tuning_tcp_fastopen: 3
net_tuning_tcp_tw_reuse: 1

# busy_poll (microseconds, only for supported drivers)
net_tuning_busy_poll: 50
net_tuning_busy_read: 50
net_tuning_busy_poll_drivers:
  - igb
  - ixgbe
  - i40e
  - ice
  - bnxt_en
  - mlx5_core
  - ena

Example Playbook

- hosts: servers
  become: true
  roles:
    - net_tuning

Example results

8 CPU, tg3, 1Gbps, 2q:        RSS mode, budget=512, conntrack=524K
8 CPU, igb, 1Gbps, 8q:        RSS mode, budget=512, conntrack=524K, busy_poll=50
20 CPU, r8169, 1Gbps, 1q:     RSS mode, budget=1280, conntrack=655K
32 CPU, igb, 1Gbps, 2q:       RPS mode (ratio 16:1), budget=2048, conntrack=1M
96 CPU, bnxt_en, 1Gbps, 32q:  RSS mode, budget=4096, conntrack=3.1M
96 CPU, bnxt_en, 10Gbps, 24q: RSS mode, budget=4096, conntrack=3.1M

Persistence

• NIC settings (RPS/RFS/XPS/queues/ring buffers) applied via systemd oneshot service on every boot
• Sysctl values persisted to /etc/sysctl.d/
• Conntrack hashsize persisted to /etc/modprobe.d/nf_conntrack.conf

License

MIT