Skip to content

perf(net): pool-back non-inline TCP host→guest data frames in poll_fast_path #263

@AprilNEA

Description

@AprilNEA

Problem

tcp_bridge.rs::poll_fast_path (called once per tick per established connection not yet promoted to inline) does:

  1. stream.read(&mut conn.read_buf) — fine, per-connection reusable buffer.
  2. ethernet::build_tcp_data_frame(&params, payload) — allocates a fresh Vec<u8> sized ETH+IP+TCP header + payload length, memcpy's headers and payload in.
  3. Pushes the Vec<u8> through a tokio mpsc Sender<Vec<u8>> to the socketpair write task.

At line rate this is one allocation + one payload memcpy per frame. Non-trivial under bulk bursts (the regressions b9f69aa and 55763d7 were partially caused by related buffer pressure).

Proposed fix

Option A — use FrameBuf::from_pool from virt/arcbox-net/src/datapath/pool.rs:

  • Change the channel type to Sender<FrameBuf> (or reuse the existing FrameBuf plumbing already used for RX).
  • build_tcp_data_frame variant that writes into a &mut [u8] provided by the pool slot, instead of returning Vec<u8>.
  • Fall back to Vec when the pool is exhausted (FrameBuf::Heap variant already handles this).

Option B — skip the channel: if the socketpair write path is cheap enough, call fd_write directly under the tokio task lock, eliminating the intermediate allocation entirely.

Prefer A — it's a drop-in replacement and keeps the write path asynchronous.

Files

  • virt/arcbox-net/src/darwin/tcp_bridge.rs::poll_fast_path
  • virt/arcbox-net/src/ethernet.rs::build_tcp_data_frame + build_tcp_data_frame_partial_csum
  • virt/arcbox-net/src/datapath/frame_buf.rs (existing pool plumbing)
  • virt/arcbox-net/src/darwin/datapath_loop.rs::send_to_guest (channel consumer)

Obviated by ABX-368

If every connection is promoted to the inline inject path (ABX-368), this whole code path goes away. Treat this as a near-term optimization if ABX-368 is blocked; otherwise prefer landing ABX-368 first.

Acceptance

  • No Vec::new / Vec::with_capacity on the host→guest TCP data path for pool-hit frames.
  • cargo test -p arcbox-net green.
  • Bulk iperf3 -R still ≥ current baseline.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions