Problem
tcp_bridge.rs::poll_fast_path (called once per tick per established connection not yet promoted to inline) does:
stream.read(&mut conn.read_buf) — fine, per-connection reusable buffer.
ethernet::build_tcp_data_frame(¶ms, payload) — allocates a fresh Vec<u8> sized ETH+IP+TCP header + payload length, memcpy's headers and payload in.
- Pushes the
Vec<u8> through a tokio mpsc Sender<Vec<u8>> to the socketpair write task.
At line rate this is one allocation + one payload memcpy per frame. Non-trivial under bulk bursts (the regressions b9f69aa and 55763d7 were partially caused by related buffer pressure).
Proposed fix
Option A — use FrameBuf::from_pool from virt/arcbox-net/src/datapath/pool.rs:
- Change the channel type to
Sender<FrameBuf> (or reuse the existing FrameBuf plumbing already used for RX).
build_tcp_data_frame variant that writes into a &mut [u8] provided by the pool slot, instead of returning Vec<u8>.
- Fall back to
Vec when the pool is exhausted (FrameBuf::Heap variant already handles this).
Option B — skip the channel: if the socketpair write path is cheap enough, call fd_write directly under the tokio task lock, eliminating the intermediate allocation entirely.
Prefer A — it's a drop-in replacement and keeps the write path asynchronous.
Files
virt/arcbox-net/src/darwin/tcp_bridge.rs::poll_fast_path
virt/arcbox-net/src/ethernet.rs::build_tcp_data_frame + build_tcp_data_frame_partial_csum
virt/arcbox-net/src/datapath/frame_buf.rs (existing pool plumbing)
virt/arcbox-net/src/darwin/datapath_loop.rs::send_to_guest (channel consumer)
If every connection is promoted to the inline inject path (ABX-368), this whole code path goes away. Treat this as a near-term optimization if ABX-368 is blocked; otherwise prefer landing ABX-368 first.
Acceptance
- No
Vec::new / Vec::with_capacity on the host→guest TCP data path for pool-hit frames.
cargo test -p arcbox-net green.
- Bulk iperf3 -R still ≥ current baseline.
Problem
tcp_bridge.rs::poll_fast_path(called once per tick per established connection not yet promoted to inline) does:stream.read(&mut conn.read_buf)— fine, per-connection reusable buffer.ethernet::build_tcp_data_frame(¶ms, payload)— allocates a freshVec<u8>sized ETH+IP+TCP header + payload length, memcpy's headers and payload in.Vec<u8>through a tokio mpscSender<Vec<u8>>to the socketpair write task.At line rate this is one allocation + one payload memcpy per frame. Non-trivial under bulk bursts (the regressions
b9f69aaand55763d7were partially caused by related buffer pressure).Proposed fix
Option A — use
FrameBuf::from_poolfromvirt/arcbox-net/src/datapath/pool.rs:Sender<FrameBuf>(or reuse the existingFrameBufplumbing already used for RX).build_tcp_data_framevariant that writes into a&mut [u8]provided by the pool slot, instead of returningVec<u8>.Vecwhen the pool is exhausted (FrameBuf::Heapvariant already handles this).Option B — skip the channel: if the socketpair write path is cheap enough, call
fd_writedirectly under the tokio task lock, eliminating the intermediate allocation entirely.Prefer A — it's a drop-in replacement and keeps the write path asynchronous.
Files
virt/arcbox-net/src/darwin/tcp_bridge.rs::poll_fast_pathvirt/arcbox-net/src/ethernet.rs::build_tcp_data_frame+build_tcp_data_frame_partial_csumvirt/arcbox-net/src/datapath/frame_buf.rs(existing pool plumbing)virt/arcbox-net/src/darwin/datapath_loop.rs::send_to_guest(channel consumer)Obviated by ABX-368
If every connection is promoted to the inline inject path (ABX-368), this whole code path goes away. Treat this as a near-term optimization if ABX-368 is blocked; otherwise prefer landing ABX-368 first.
Acceptance
Vec::new/Vec::with_capacityon the host→guest TCP data path for pool-hit frames.cargo test -p arcbox-netgreen.