Perf/shm latency and compiler optimizations (#116)

* perf: SHM latency optimizations — close gap with reference C implementation

- Add .cargo/config.toml with target-cpu=native for optimal codegen
- Replace nix crate clock_gettime wrapper with direct libc call
- Capture receive timestamp inside receive_blocking() immediately after
  condvar wake, matching C measurement point
- Eliminate redundant zero-fill: vec![0u8;N] → Vec::with_capacity + set_len
- Replace per-byte ring buffer copies with copy_nonoverlapping in blocking path
- Add #[inline] hints to all hot-path SHM functions
- Server loop uses transport-captured timestamp when available

Reduces SHM direct mode mean latency by ~12.5% (23.4µs → 20.5µs),
narrowing gap vs reference C benchmark from 25% to ~10%.

Co-authored-by: Cursor <cursoragent@cursor.com>

* perf: clarify receive timestamp comment in SHM-direct

Update comment on the receive-side clock_gettime call to better
describe why it's captured inside the mutex (matches reference C
approach for accurate latency measurement).

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs: add detailed PERF comments to all optimized code paths

Document the rationale behind each optimization with inline comments:
- .cargo/config.toml: explain target-cpu=native and portability note
- mod.rs: explain direct libc vs nix crate clock_gettime, receive_time_ns field
- shared_memory_direct.rs: send/receive timestamp placement, zero-fill elimination
- shared_memory_blocking.rs: bulk copy_nonoverlapping vs byte-by-byte with before/after
- shared_memory.rs: inline hints on ring buffer hot-path functions
- main.rs: transport-level timestamp preference in both server loops

Co-authored-by: Cursor <cursoragent@cursor.com>

* style: fix cargo fmt formatting issues

- Collapse short copy_nonoverlapping calls to single line in shared_memory_blocking.rs
- Remove extra blank line in shared_memory_direct.rs

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: address PR review — conditional timestamp placement and clock_gettime error handling

Move SHM-direct receive timestamp inside/outside mutex based on
--send-delay: latency benchmarks (send-delay > 0) capture inside the
mutex for accuracy matching the reference C implementation; throughput
benchmarks (no send-delay) capture after mutex unlock to eliminate the
22-31% regression at small message sizes. The flag is derived
automatically with no new user-facing CLI options.

Add debug_assert! on all raw clock_gettime return values as cheap
insurance against silent failures.

Remove .cargo/config.toml (target-cpu=native) to restore binary
portability across CPU variants.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs+test: add precise_timestamps tests, CPU-optimized build docs, and target-cpu=native rationale

- Add 3 unit tests for BlockingSharedMemoryDirect::with_precise_timestamps():
  constructor flag verification (true/false) and end-to-end receive with
  precise_timestamps=true exercising the inside-mutex timestamp code path
- Add factory test verifying send_delay variants (None, ZERO, 10ms) are
  accepted when creating SHM-direct transports
- Document SHM-direct conditional timestamp placement in README: adaptive
  inside/outside-mutex receive timestamp based on --send-delay, with
  latency vs throughput tradeoff explanation (22-31% regression context)
- Document CPU-optimized builds in README: rationale for removing
  .cargo/config.toml (portability, CI cross-compilation risks across
  NXP S32G/Qualcomm Ride SX4/Renesas R-Car S4), on-target builds with
  RUSTFLAGS="-C target-cpu=native", per-platform cross-compile examples
- Update CONFIG.md Rust Compiler Optimizations section with callout
  explaining why target-cpu=native must not be in repo-wide config,
  add cross-compile example and link to README
- Fix pre-existing clippy lint: map_or -> is_some_and on send_delay wiring
- All tests passing, clippy clean, cargo fmt applied

AI-assisted-by: Claude Opus 4 (Anthropic)

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: mcurrier2

CONFIG.md

@@ -334,10 +334,26 @@ sudo sysctl -p
 ### Application-Level Tuning
 
 #### Rust Compiler Optimizations
+
+> **Important:** This repo does **not** ship a `.cargo/config.toml` with
+> `target-cpu=native`. That setting would apply to every build —
+> including CI — producing non-portable binaries. When CI
+> cross-compiles for ARM on x86 runners, `target-cpu=native`
+> silently optimizes for the build host, not the target, and may
+> emit instructions the deployment CPU does not support (e.g.,
+> ARMv8.2+ on a Cortex-A53). Apply CPU tuning explicitly at build
+> time instead. See the README's
+> [CPU-Optimized Builds](README.md#cpu-optimized-builds) section
+> for per-platform examples.
+
 ```bash
-# Maximum optimization
+# On-target build (uses every instruction the local CPU supports)
 RUSTFLAGS="-C target-cpu=native -C opt-level=3" cargo build --release
 
+# Cross-compile for a specific ARM CPU
+RUSTFLAGS="-C target-cpu=cortex-a53 -C opt-level=3" cargo build \
+  --release --target aarch64-unknown-linux-gnu
+
 # Link-time optimization
 RUSTFLAGS="-C lto=fat" cargo build --release
 

README.md

@@ -205,6 +205,16 @@ send-wait-receive per message).
 3. **Client Side**: Timestamp captured after receiving response
 4. **Latency Calculation**: Total elapsed time from send to receive
 
+#### SHM-Direct Conditional Timestamp Placement
+
+The `--shm-direct` transport uses **adaptive receive timestamp placement** based on the test type, controlled automatically by `--send-delay`:
+
+- **Latency-focused tests** (`--send-delay > 0`): The receive timestamp is captured **inside the mutex**, immediately after the condvar wake-up. This matches the reference C SHM implementation and excludes payload copy, allocation, and mutex unlock from measured latency (~5–10 µs savings). The send-delay between messages dwarfs any additional mutex contention.
+
+- **Throughput-focused tests** (no `--send-delay`): The receive timestamp is captured **after the mutex unlock**, keeping the critical section minimal. This avoids a 22–31% throughput regression at small message sizes caused by the extra `clock_gettime` call inside the mutex.
+
+This behavior is fully automatic — no additional CLI flags are needed. The `--send-delay` flag is sufficient to signal intent: if you're pacing messages for latency measurement, you get the most accurate timestamps; if you're saturating the pipe for throughput, you get maximum performance.
+
 #### Streaming Output Columns
 
 The per-message streaming output (JSON and CSV) contains the
@@ -358,6 +368,37 @@ cargo build --release
 
 The optimized binary will be available at `target/release/ipc-benchmark`.
 
+### CPU-Optimized Builds
+
+By default, `cargo build --release` produces **portable binaries** that run on any CPU in the target architecture family (e.g., generic `aarch64`). This is intentional — the repo does not ship a `.cargo/config.toml` with `target-cpu=native` because that setting would silently affect every build, including CI, producing non-portable binaries that may use instructions unsupported on the deployment target.
+
+This matters especially for cross-platform ARM development. If CI runs on AWS Graviton (Neoverse-N1) but the target is an NXP S32G (Cortex-A53), a `target-cpu=native` binary built on Graviton could use ARMv8.2+ instructions that the Cortex-A53 does not support, causing illegal-instruction crashes at runtime.
+
+**When building directly on target hardware**, enable CPU-specific optimizations at build time:
+
+```bash
+# On-target build: let the compiler use every instruction the local CPU supports
+RUSTFLAGS="-C target-cpu=native" cargo build --release
+```
+
+**When cross-compiling in CI**, specify the exact CPU target per platform:
+
+```bash
+# NXP S32G (Cortex-A53)
+RUSTFLAGS="-C target-cpu=cortex-a53" cargo build --release \
+  --target aarch64-unknown-linux-gnu
+
+# Qualcomm Ride SX4 (Cortex-A78AE) — use the closest supported LLVM target
+RUSTFLAGS="-C target-cpu=cortex-a78" cargo build --release \
+  --target aarch64-unknown-linux-gnu
+
+# Renesas R-Car S4 (Cortex-A76)
+RUSTFLAGS="-C target-cpu=cortex-a76" cargo build --release \
+  --target aarch64-unknown-linux-gnu
+```
+
+The performance-critical code paths in this project (timestamp placement, bulk copies, zero-fill elimination, direct `libc::clock_gettime`) are pure code optimizations that do not depend on `target-cpu`. They provide the bulk of the latency improvement regardless of CPU target. The `target-cpu` flag adds a smaller, incremental gain from SIMD auto-vectorization and instruction scheduling tuned for the specific microarchitecture.
+
 ### Quick Start
 
 ```bash

src/benchmark_blocking.rs

@@ -477,6 +477,13 @@ impl BlockingBenchmarkRunner {
                 .arg(self.config.pmq_priority.to_string());
         }
 
+        // Forward send-delay to server so SHM-direct can enable precise
+        // (inside-mutex) timestamps for latency-focused benchmarks.
+        if let Some(delay) = self.config.send_delay {
+            let micros = delay.as_micros();
+            cmd.arg("--send-delay").arg(format!("{micros}us"));
+        }
+
         // Add latency file path if provided (for true IPC measurement)
         if let Some(path) = latency_file_path {
             cmd.arg("--internal-latency-file").arg(path);
@@ -813,8 +820,11 @@ impl BlockingBenchmarkRunner {
     /// - `Ok(())`: Warmup completed successfully
     /// - `Err(anyhow::Error)`: Warmup failed
     fn run_warmup(&self, transport_config: &TransportConfig) -> Result<()> {
-        let mut client_transport =
-            BlockingTransportFactory::create(&self.mechanism, self.args.shm_direct)?;
+        let mut client_transport = BlockingTransportFactory::create(
+            &self.mechanism,
+            self.args.shm_direct,
+            self.config.send_delay,
+        )?;
 
         // --- Server Process Spawning ---
         let (mut server_process, mut pipe_reader) = self.spawn_server_process(transport_config)?;
@@ -1000,8 +1010,11 @@ impl BlockingBenchmarkRunner {
         metrics_collector: &mut MetricsCollector,
         mut results_manager: Option<&mut crate::results_blocking::BlockingResultsManager>,
     ) -> Result<()> {
-        let mut client_transport =
-            BlockingTransportFactory::create(&self.mechanism, self.args.shm_direct)?;
+        let mut client_transport = BlockingTransportFactory::create(
+            &self.mechanism,
+            self.args.shm_direct,
+            self.config.send_delay,
+        )?;
 
         // Create a temporary file for server to write latencies
         let latency_file_path = std::env::temp_dir()
@@ -1181,8 +1194,11 @@ impl BlockingBenchmarkRunner {
         metrics_collector: &mut MetricsCollector,
         mut results_manager: Option<&mut crate::results_blocking::BlockingResultsManager>,
     ) -> Result<()> {
-        let mut client_transport =
-            BlockingTransportFactory::create(&self.mechanism, self.args.shm_direct)?;
+        let mut client_transport = BlockingTransportFactory::create(
+            &self.mechanism,
+            self.args.shm_direct,
+            self.config.send_delay,
+        )?;
 
         // --- Server Process Spawning ---
         let (mut server_process, mut pipe_reader) = self.spawn_server_process(transport_config)?;