Expose driver pipeline depth as a benchpress job parameter

Reviewed By: YifanYuan3

Differential Revision: D113146808

Author: excelle08

benchpress/config/jobs.yml

@@ -502,6 +502,7 @@
     - '--rpc-fanout-scale={rpc_fanout_scale}'
     - '--server-zstd={server_zstd}'
     - '--sla-p95-ms={sla_p95_ms}'
+    - '--depth={depth}'
     - '{extra_args}'
   vars:
     # Hardcode num_instances=1: this job exists specifically for single-instance
@@ -534,6 +535,12 @@
     - 'rpc_fanout_scale=0.05'
     - 'server_zstd=0'
     - 'sla_p95_ms=700'
+    # Driver pipeline depth (max outstanding requests per driver connection).
+    # Default 1. Raise (e.g. 2) if the final phase saturates neither CPU nor SLA
+    # latency (final p95 well below sla_p95_ms while CPU util < ~90%) — often
+    # needed on high-perf ARM cores. With adaptive depth on (the default), this is
+    # the starting floor the peak search raises from; see README "Driver depth".
+    - 'depth=1'
     - 'extra_args='
   hooks:
     - hook: cpu-mpstat
@@ -592,6 +599,7 @@
     - '--rpc-fanout-scale={rpc_fanout_scale}'
     - '--server-zstd={server_zstd}'
     - '--sla-p95-ms={sla_p95_ms}'
+    - '--depth={depth}'
     - '{extra_args}'
   vars:
     - 'num_instances=-1'
@@ -635,6 +643,12 @@
     - 'rpc_fanout_scale=0.05'
     - 'server_zstd=0'
     - 'sla_p95_ms=700'
+    # Driver pipeline depth (max outstanding requests per driver connection).
+    # Default 1. Raise (e.g. 2) if the final phase saturates neither CPU nor SLA
+    # latency (final p95 well below sla_p95_ms while CPU util < ~90%) — often
+    # needed on high-perf ARM cores. With adaptive depth on (the default), this is
+    # the starting floor the peak search raises from; see README "Driver depth".
+    - 'depth=1'
     - 'extra_args='
   hooks:
     - hook: cpu-mpstat
@@ -695,6 +709,7 @@
     - '--rpc-fanout-scale={rpc_fanout_scale}'
     - '--server-zstd={server_zstd}'
     - '--sla-p95-ms={sla_p95_ms}'
+    - '--depth={depth}'
     - '{extra_args}'
   vars:
     - 'num_instances=-1'
@@ -735,6 +750,12 @@
     - 'rpc_fanout_scale=0.05'
     - 'server_zstd=0'
     - 'sla_p95_ms=700'
+    # Driver pipeline depth (max outstanding requests per driver connection).
+    # Default 1. Raise (e.g. 2) if the final phase saturates neither CPU nor SLA
+    # latency (final p95 well below sla_p95_ms while CPU util < ~90%) — often
+    # needed on high-perf ARM cores. With adaptive depth on (the default), this is
+    # the starting floor the peak search raises from; see README "Driver depth".
+    - 'depth=1'
     - 'extra_args='
   hooks:
     - hook: copymove

packages/feedsim/README.md

@@ -55,7 +55,7 @@ taskset -c 0-15 ./benchpress_cli.py install feedsim_dlrm
 Unlike FeedSim v1 which spawns a new FeedSim instance per 100 CPU cores,
 `feedsim_dlrm` is pinned to **one FeedSim instance per host** because the
 redesigned threading model in FeedSim v2 has overcome the scalability issue
-on ultra-high-core-count CPUs and ARM CPUs. 
+on ultra-high-core-count CPUs and ARM CPUs.
 
 The runner searches for the QPS that keeps 95th-percentile end-to-end
 latency at or below **700 ms**. When it converges it runs a final 5-minute
@@ -64,7 +64,7 @@ counters) during that final window. We expect the total wall-clock runtime
 to be around 30 minutes.
 
 Please make sure to turn CPU turbo-boost on before starting, or FeedSim may
-fail to converge and report a low QPS. 
+fail to converge and report a low QPS.
 
 ### Result report
 
@@ -211,7 +211,7 @@ feedsim server, driver and mock_services instances.
 
 2. Use the `feedsim_autoscale_dlrm` job. This autoscale job will spawn `ceil(nproc / 100)`
 FeedSim instances, each pinned to its own CPU range via `taskset`, plus one driver
-and one `mock_services` process per instance (also `taskset`-isolated). For example: 
+and one `mock_services` process per instance (also `taskset`-isolated). For example:
 ```
 ./benchpress_cli.py run feedsim_autoscale_dlrm
 ```
@@ -220,6 +220,41 @@ In multi-instance mode, the overall QPS is the sum across all instances. and the
 average latency will be the average of p95 latency values observed across all
 instances.
 
+### Driver depth (fixing CPU/latency under-utilization)
+
+The `depth` parameter sets the driver's pipeline depth — the maximum number of
+outstanding (in-flight) requests per driver connection. The driver's total
+offered concurrency is `driver_threads × connections × depth`, so with the
+default `depth=1` the driver can cap the achievable load below what the server
+can actually handle.
+
+**Increase `depth` beyond 1 when the final benchmarking phase saturates neither
+CPU nor latency** — i.e. the final achieved p95 latency is well below the SLA
+limit (`sla_p95_ms`, default 700 ms) *and* the CPU utilization during the final
+5-minute benchmarking phase is less than ~90%. In that situation the reported QPS
+is limited by driver concurrency rather than by the server, so it understates the
+hardware's true capacity. Raising `depth` (start with `2`) lets the driver offer
+more concurrent load until the server becomes the bottleneck — either CPU-bound
+(~100% utilization) or latency-bound (p95 ≈ SLA). **This is likely necessary on
+high-performance ARM cores** (e.g. NVIDIA Grace), which can otherwise sit at
+80–90% CPU with p95 far below the SLA at `depth=1`.
+
+```
+# Force driver depth 2
+./benchpress_cli.py run feedsim_dlrm -i '{"depth": 2}'
+```
+
+There is also an **adaptive depth** mechanism (on by default) that raises the
+depth automatically during the peak-finding stage until the server saturates
+(system CPU ≥ 95% or p95 ≥ SLA). It catches *severe* under-utilization early, but
+because it evaluates saturation on the high-load peak/search probes rather than
+on the final SLA-converged operating point, it **may not catch all
+under-utilization cases**. If you still observe under-utilization in the final
+result (low CPU + p95 well under SLA), increase `depth` manually as above. When
+adaptive depth is on, a manually-set `depth` acts as the starting floor the
+adaptive search raises from; to pin an exact fixed depth, also set the
+`FEEDSIM_ADAPTIVE_DEPTH_MAX=0` environment variable to disable adaptive search.
+
 ### Other parameters
 
 This section lists additional parameters in `feedsim_dlrm` benchmark. These parameters
@@ -233,6 +268,7 @@ Job-level parameters (can be passed via `-i` flag in Benchpress CLI):
 |---|---|---|
 | `num_instances` | Number of FeedSim instances to run in parallel. Defaults to 1 in `feedsim_dlrm`; set to -1 to autoscale for `feedsim_autoscale_dlrm`. | `1` |
 | `sla_p95_ms` | SLA target in ms. The runner searches for the highest QPS keeping p95 ≤ this. | `700` |
+| `depth` | Driver pipeline depth (max outstanding requests per connection; total in-flight = `driver_threads × connections × depth`). Raise (e.g. `2`) when the final phase saturates neither CPU nor latency — often needed on high-perf ARM. See [Driver depth](#driver-depth-fixing-cpulatency-under-utilization). | `1` |
 | `io_dist` | I/O latency distribution: `fixed`, `exponential`, or `lognormal`. | `fixed` |
 | `io_mean` | Mean I/O latency in ms. | `200` |
 | `workload` | Ranking workload: `pagerank` or `dlrm`. `dlrm` is v2. | `dlrm` |

packages/feedsim/run.sh

@@ -120,6 +120,7 @@ Usage: ${0##*/} [OPTION]...
     --rpc-fanout-scale Scale factor applied to per-session fanout counts. Default: 0.05 (t43 c7).
     --server-zstd Enable ZSTD compression on server-side response payloads (0=off, 1=on). Default: 0 (t43 c7).
     --sla-p95-ms search_qps SLA target (95th percentile latency in ms). Default: 700.
+    --depth Driver pipeline depth: max outstanding requests per driver connection (max in-flight = driver_threads * connections * depth). Default: 1 (or \$FEEDSIM_DRIVER_DEPTH). Raise (e.g. 2) when the final phase saturates neither CPU nor SLA latency; with adaptive depth on, this is the starting floor the peak search raises from.
 EOF
 }
 
@@ -316,6 +317,12 @@ main() {
     local sla_p95_ms
     sla_p95_ms="700"
 
+    # Driver pipeline depth (max outstanding requests per driver connection).
+    # Env var FEEDSIM_DRIVER_DEPTH is the fallback default; the --depth CLI flag
+    # (forwarded from the benchpress `depth` job parameter) overrides it.
+    local driver_depth
+    driver_depth="${FEEDSIM_DRIVER_DEPTH:-1}"
+
     if [ -z "$IS_AUTOSCALE_RUN" ]; then
        echo > $BREPS_LFILE
     fi
@@ -644,6 +651,13 @@ main() {
             --sla-p95-ms=*)
                 sla_p95_ms="${1#*=}"
                 ;;
+            --depth)
+                driver_depth="$2"
+                shift
+                ;;
+            --depth=*)
+                driver_depth="${1#*=}"
+                ;;
             -h|--help)
                 show_help >&2
                 exit 1
@@ -1002,7 +1016,9 @@ main() {
     # to reach a real bound instead of capping on driver concurrency (the t19
     # anti-pattern where big boxes sat at ~80% CPU with p95 far below SLA).
     # Enabled by default up to depth 8; set FEEDSIM_ADAPTIVE_DEPTH_MAX=0 to
-    # disable and fall back to the fixed FEEDSIM_DRIVER_DEPTH (default 1).
+    # disable and use the fixed driver_depth (--depth flag / FEEDSIM_DRIVER_DEPTH,
+    # default 1). When adaptive is on, driver_depth is the STARTING floor the peak
+    # search raises from (search_qps reads the --depth we pass below).
     sqps_adaptive_arg=""
     adaptive_depth_max="${FEEDSIM_ADAPTIVE_DEPTH_MAX:-8}"
     if [ "$adaptive_depth_max" != "0" ]; then
@@ -1019,7 +1035,7 @@ main() {
                 --monitor_port "$client_monitor_port" \
                 --threads="${driver_threads}" \
                 --connections=4 \
-                --depth="${FEEDSIM_DRIVER_DEPTH:-1}" \
+                --depth="${driver_depth}" \
                 $client_feature_opts \
                 $silesia_opts \
                 $req_size_opts
@@ -1032,7 +1048,7 @@ main() {
             $driver_bin \
                 --monitor_port "$client_monitor_port" \
                 --server "0.0.0.0:$port" \
-                --depth="${FEEDSIM_DRIVER_DEPTH:-1}" \
+                --depth="${driver_depth}" \
                 $client_feature_opts \
                 $silesia_opts \
                 $req_size_opts
@@ -1061,7 +1077,7 @@ main() {
                 --monitor_port "$client_monitor_port" \
                 --threads="${num_workers}" \
                 --connections="${num_connections}" \
-                --depth="${FEEDSIM_DRIVER_DEPTH:-1}" \
+                --depth="${driver_depth}" \
                 $client_feature_opts \
                 $silesia_opts \
                 $req_size_opts

packages/feedsim/third_party/src/scripts/search_qps.sh

@@ -120,10 +120,11 @@ mutilate (EuroSys \'14) [https://github.com/leverich/mutilate]
     -P          PID of the process to log runtime breakdowns. Optional
     -B          Folder to log runtime breakdowns. Optional
     -D          Adaptive depth: max driver pipeline depth. When set, the peak
-                phase raises the driver's --depth from 1 until the server is
-                saturated (system CPU >= 95% OR achieved p95 >= SLA), then holds
-                that depth for the QPS search. Overrides any --depth in the
-                driver command. Optional.
+                phase raises the driver's --depth until the server is saturated
+                (system CPU >= 95% OR achieved p95 >= SLA), then holds that depth
+                for the QPS search. The search STARTS from any --depth in the
+                driver command (default 1), so a manually-set --depth acts as a
+                floor. Optional.
 EOF
 }
 
@@ -394,12 +395,18 @@ fi
 # remaining argument is loadtest command
 command=$@
 
-# In adaptive-depth mode, search_qps owns the driver's --depth: strip any fixed
-# --depth from the command so our per-attempt --depth is the only one, and start
-# the peak search at depth=1.
+# In adaptive-depth mode, search_qps owns the driver's --depth: capture any fixed
+# --depth as the STARTING depth (so a manually-set --depth acts as a floor the
+# peak search raises from), then strip it so our per-attempt --depth is the only
+# one on the command. Default start is depth=1 (unchanged behavior).
+adaptive_start_depth=1
 if [ -n "$adaptive_depth_max" ]; then
+  fixed_depth=$(echo "$command" | grep -oE -- '--depth=[0-9]+' | head -1 | grep -oE '[0-9]+')
+  if [ -n "$fixed_depth" ] && [ "$fixed_depth" -gt 1 ]; then
+    adaptive_start_depth=$fixed_depth
+  fi
   command=$(echo "$command" | sed -E 's/[[:space:]]*--depth=[0-9]+//g')
-  adaptive_depth_arg="--depth=1"
+  adaptive_depth_arg="--depth=$adaptive_start_depth"
 fi
 
 # make sure latency_type and latency_target are specified
@@ -553,12 +560,14 @@ fi
 benchreps_tell_state "before peak_qps"
 if [ -n "$adaptive_depth_max" ]; then
   # Adaptive depth: the peak load test offers at most threads*connections*depth
-  # concurrent requests. Starting at depth=1, keep raising depth (and re-running
-  # peak) until the server saturates — system CPU >= 95% OR achieved p95 >= SLA
+  # concurrent requests. Starting at adaptive_start_depth, keep raising depth (and
+  # re-running peak) until the server saturates — system CPU >= 95% OR p95 >= SLA
   # — so platforms that need more offered concurrency reach a real bound instead
   # of capping on driver concurrency (the t19 anti-pattern). The selected depth
-  # is then held for the QPS search / tuning / final phases.
-  cur_depth=1
+  # is then held for the QPS search / tuning / final phases. The starting depth
+  # is the fixed --depth from the driver command (default 1), so a manually-set
+  # depth raises the floor.
+  cur_depth=$adaptive_start_depth
   while : ; do
     adaptive_depth_arg="--depth=$cur_depth"
     # Sample system CPU busy% over a mid-run window while the peak load runs.