docs: Update benchmark scenarios matrix and integration strategy #381
+226
−104
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,110 +4,139 @@ latency of model-serving pods within the LLM-D Fast Model Actuation workflow. | |
|
|
||
| ## Purpose | ||
| The goal is to quantify and compare how quickly a model-serving duo (server-requesting | ||
| and server-providing pods), when integrated with the Workload Variant Autoscaler (WVA), | ||
| becomes available under three different actuation conditions in order of decreasing | ||
| latency: | ||
|
|
||
| - **Cold start**: creating a new vLLM instance without using a launcher | ||
| - **Warm start**: creating a new vLLM instance in an existing launcher pod | ||
| - **Hot start**: waking a sleeping vLLM instance on an existing launcher pod | ||
|
|
||
| These metrics will guide future optimizations for the **Dual-Pods Controller (DPC)**. Ultimately, the goal | ||
| is *high predictability*, which is defined as achieving close to 100% hit rate of awakening | ||
| available, sleeping pods on cluster GPUs as function of total inference server | ||
| requests for common user scenarios. | ||
|
|
||
| ## Measurement Layers | ||
|
|
||
| FMA benchmarking uses a layered measurement model. Layer 1 is what FMA benchmarks own | ||
| directly. Layer 2 bridges actuation to inference readiness (i.e., latency for inference | ||
| requests to get responses back in an FMA-enabled context). Layer 3 is out of FMA's | ||
| direct scope but is referenced for completeness and handoff to other frameworks. | ||
|
|
||
| | Layer | Focus | Metrics | Measured By | | ||
| | ----- | ----- | ------- | ----------- | | ||
| | **L1: Actuation** | Requester pod readiness | T_actuation (requester creation to readiness), T_wake (DPC wakes sleeping vLLM instance), Hit_rate (GPU hits), T_launcher (launcher creates new vLLM instance) | llm-d-benchmark new harness | | ||
| | **L2: Inference Readiness** | First inference response | T_e2e (requester creation to first inference response), T_first_token (requester ready to first inference response) | llm-d-benchmark nop/inference-perf harness | | ||
| | **L3: Steady-State** | Throughput/latency | T_actuation (requester creation to readiness), TPOT (time per output token), throughput, queue depth, KV cache usage, replica stability | llm-d-benchmark / WVA | | ||
|
Collaborator
There was a problem hiding this comment. I find L3 surprising and confusing. The difference from L1 to L2 is not like the difference from L2 to L3. The difference from L1 to L2 is more of the same thing: more latency (picking a later event that stops the clock). I expected L3 to differ in the same way; perhaps start the clock earlier (e.g., when an inference client sends a request). But mostly L3 is about different stuff, not latency. I do not see how WVA plays a role in any of the things listed for L3 (nor L2 nor L1), which is also a surprise. I expected to see WVA involved somewhere.
Collaborator
Author
There was a problem hiding this comment. L3 is intentionally different in kind from L1/L2. L1 and L2 are stop-the-clock latency measurements that FMA owns. L3 is about steady-state performance metrics (TPOT, throughput, queue depth, KV cache usage, replica stability) that become relevant when FMA is integrated with WVA and the broader llm-d stack. The inclusion of T_actuation in L3 is the bridge: it lets WVA and llm-d-benchmark consumers see how FMA actuation latency affects their steady-state metrics. The pitch is: look at how your metrics perform with FMA included. I had Claude double check that these metrics are currently being captured: TPOT, throughput, KV cache usage, and queue depth are in llm-d-benchmark's schema v0.2 and
Collaborator
There was a problem hiding this comment. Why is there one and only one metric that appears in multiple layers?
Collaborator
Author
There was a problem hiding this comment. As stated in my previous comment, T_actuation is the bridge that we care most about. It'''s the one metric that FMA uniquely contributes to the L3 picture, letting WVA and llm-d-benchmark consumers see how actuation latency affects their steady-state metrics. The other L1/L2 metrics are either FMA-internal (T_wake, Hit_rate, T_launcher) or already captured by L3 tools in their own terms (TTFT maps to T_first_token). |
||
|
|
||
| **Metric definitions:** | ||
|
|
||
| - **T_actuation**: Time from requester pod creation (ReplicaSet scale-up) to requester pod readiness (`/ready` probe passes), which implies the DPC has bound the requester to a server-providing pod and the vLLM instance is serving. | ||
| - **T_wake**: Request-response time for the DPC's `/wake_up` call to a sleeping vLLM instance on the server-providing pod. A part of T_actuation when a hot start occurs. | ||
| - **Hit_rate**: Fraction of requesters that get bound to an existing sleeping pod on the correct GPU (hit) vs. requiring a cold start (i.e., new vLLM instance in existing launcher pod or new launcher pod + new vLLM instance). | ||
aavarghese marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - **T_launcher**: Time from the launcher receiving a create request to the new vLLM instance reporting healthy. Includes the benefit of vLLM module preloading. | ||
rubambiza marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - **T_e2e**: Total time from requester pod creation to first successful inference response. Spans the full path: requester scheduling, DPC binding, instance wake-up or launcher instance creation, vLLM ready, first inference (T_actuation + T_first_token). | ||
| - **T_first_token**: Time from requester pod readiness to first successful inference response received through the server-providing pod's vLLM instance (time-to-first-token, post-actuation). | ||
|
|
||
| ## Benchmarking Scenarios | ||
|
|
||
| | Scenario | Description | | ||
| | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | **Fast Replica Scale Up** | As a ModelService Owner, I can scale up the number of active replicas for a variant from 0 to 1 or 1 to N with minimal latency | | ||
| | **Introducing New Variant** | As a ModelService Owner, I can deploy a newly released variant in anticipation of user requests | | ||
| | **Resource Request Justification** | As a Workload Owner, I can stress-test my namespace's resources to justify more resource requests (routes, gateways, GPUs) from cluster operator | | ||
| | **Maintenance Planning** | As a Cluster Operator, I can validate the cluster performance is similar or better after node maintenance schedules and upgrades | | ||
|
|
||
|
|
||
| ## Benchmarking Matrix | ||
|
|
||
| ### Actuation Paths (columns) | ||
|
|
||
| The columns represent the different paths FMA can take to satisfy a new server-requesting pod, | ||
| using the team's established terminology: | ||
|
|
||
| | Actuation Path | What It Measures | llm-d-benchmark Config | | ||
| | ------------------------- | ---------------- | ---------------------- | | ||
| | **Cold Start** | No launcher, no sleeping pods. Raw Kubernetes deploy-to-ready latency (non-FMA baseline, or FMA milestone 2 without launcher). | `-t standalone` comparison baseline | | ||
| | **Warm Start** | A launcher pod exists (pre-created by the LPC) but no sleeping instance is available on the assigned GPU. Launcher creates a new vLLM instance with the benefit of module preloading. | `-t fma` with default `LLMDBENCH_FMA_LAUNCHER_*` env vars | | ||
| | **Hot Start** | A sleeping vLLM instance exists on the correct GPU. DPC sends `/wake_up`. Best-case actuation path. | `-t fma` with `LLMDBENCH_VLLM_COMMON_ENABLE_SLEEP_MODE=true`, `LLMDBENCH_FMA_DUAL_POD_SLEEPER_LIMIT>=1` | | ||
|
|
||
| > **Note on simulation:** Any of the above paths can be exercised with mock GPUs | ||
| > (`llm-d-inference-sim` image or launcher `--mock-mode`) for CI pipelines and scenario | ||
| > prototyping. Simulation is an orthogonal testing mode, not a separate actuation path. | ||
| > | ||
| > **Note on caching:** Model tensor caching (via PVC) and CUDA graph compilation caching | ||
| > are orthogonal to the actuation paths above. Either or both can be enabled for any path | ||
| > besides Hot Start (where the instance is already loaded). Caching configuration is | ||
| > controlled via `LLMDBENCH_VLLM_COMMON_EXTRA_PVC_NAME` and `LLMDBENCH_VLLM_COMMON_VLLM_CACHE_ROOT` | ||
| > in the `fma.sh` scenario. | ||
|
|
||
| ### Matrix | ||
|
|
||
| Cell annotations indicate which measurement layers apply: | ||
| - **L1** -- Layer 1 actuation metrics (T_actuation, T_wake, Hit_rate, T_launcher) | ||
| - **L1+L2** -- Actuation metrics plus inference readiness (T_first_token, T_e2e) | ||
| - **P** -- Planned but not yet implemented | ||
| - **--** -- Not applicable to this combination | ||
|
|
||
| | Scenario | Cold Start | Warm Start | Hot Start | | ||
| | ---------------------------------- | :--------: | :--------: | :-------: | | ||
| | **Fast Replica Scale Up** | L1+L2 | L1+L2 | L1+L2 | | ||
| | **Introducing New Variant** | L1+L2 | L1+L2 | -- | | ||
| | **Resource Request Justification** | L1 | L1 | L1 | | ||
| | **Maintenance Planning** | L1+L2 | L1+L2 | L1+L2 | | ||
|
|
||
|
|
||
| ### Scenario Rationale | ||
|
|
||
| | FMA Scenario | Why Included | llm-d-benchmark Applicability | | ||
| | ---------------------------------- | ------------ | ----------------------------- | | ||
| | **Fast Replica Scale Up** | Core FMA value proposition: how fast can the DPC bring replicas online? Directly measures the benefit of sleep/wake and launcher preloading over cold starts. | `fma.sh` scenario with varying `LLMDBENCH_VLLM_COMMON_REPLICAS`; `inference-scheduling` guide with `-t fma` | | ||
| | **Introducing New Variant** | Measures actuation latency for a previously unseen model (cold cache, no sleeping instances for this model). Assumes the LPC has already created the needed launchers. Captures the "day 1" deployment experience. | `fma.sh` with `LLMDBENCH_DEPLOY_MODEL_LIST` variations; nop harness for pure actuation timing | | ||
| | **Resource Request Justification** | Stress-tests namespace resources across multiple concurrent models/variants to produce data for capacity planning and resource justification to cluster operators. | `fma.sh` with multi-model list; DoE experiment with replica/model treatments | | ||
| | **Maintenance Planning** | Regression baseline: run the same scenarios before and after node maintenance or upgrades. Detects performance regressions in actuation latency. | Any guide scenario as regression baseline with `-t fma`; compare pre/post results | | ||
|
|
||
| ### Actuation Path Rationale | ||
|
|
||
| | Actuation Path | Why Included | | ||
| | ---------------- | ------------ | | ||
| | **Cold Start** | Baseline without FMA (or FMA milestone 2 without launcher). Establishes the raw Kubernetes deploy-to-ready latency that all FMA paths should improve upon. | | ||
| | **Warm Start** | Measures the launcher's contribution when no sleeping instance is available. LPC has pre-created launcher pods, and the launcher creates a new vLLM instance with module preloading benefit. | | ||
| | **Hot Start** | Best-case FMA path. DPC sends `/wake_up` to a sleeping vLLM instance on the correct GPU. Measures sleep-to-wake latency. | | ||
|
|
||
|
|
||
| ## Integration Strategy | ||
|
|
||
| ### Current State | ||
|
|
||
| FMA is being integrated as a third deploy method (`-t fma`) in [llm-d-benchmark](https://github.com/llm-d/llm-d-benchmark), | ||
| alongside `standalone` and `modelservice`. This work is tracked on the | ||
| [`fma` branch](https://github.com/manoelmarques/llm-d-benchmark/tree/fma) and includes: | ||
|
|
||
| - **`scenarios/examples/fma.sh`** -- Scenario configuration with sleep mode enabled, model caching PVC, and FMA image references. | ||
| - **`setup/steps/07_deploy_fma_models.py`** -- Standup step that deploys InferenceServerConfig, LauncherConfig, LauncherPopulationPolicy, and requester ReplicaSet CRs. Installs FMA CRDs and the `fma-controllers` Helm chart. Waits for dual-pod controller and launcher-populator readiness. | ||
| - **`setup/env.sh`** -- 35+ new `LLMDBENCH_FMA_*` environment variables covering chart version, image registry/tags, dual-pod configuration, launcher configuration, and requester resource limits. | ||
| - **`setup/run.sh`** -- FMA endpoint discovery via Kubernetes service labels (`stood-up-via=fma`). | ||
| - **`setup/teardown.sh`** -- Ordered teardown: FMA custom resources first, then wait for the dual-pods controller to remove finalizers, then uninstall the FMA Helm release. | ||
|
|
||
| The existing llm-d-benchmark harnesses (nop, inference-perf, vllm-benchmark) can run after | ||
| FMA standup to measure L2 and L3 metrics. | ||
|
|
||
| ### Next Steps | ||
|
|
||
| 1. **Upstream the `fma` branch** into llm-d-benchmark, aligning with the declarative | ||
| Python architecture in [PR #848](https://github.com/llm-d/llm-d-benchmark/pull/848). | ||
| 2. **Add FMA-specific experiment YAML** for Design of Experiments (DoE) treatments: | ||
| replica count, sleep mode on/off, sleeper limit, model variant combinations. | ||
| 3. **Add actuation-specific metrics collection** in the nop harness: T_actuation, T_wake, | ||
| Hit_rate parsed from FMA pod events and DPC logs. | ||
| 4. **Consider Grafana integration** for visual actuation metrics (scale-up latency | ||
| dashboards), following the pattern in [WVA PR #900](https://github.com/llm-d/llm-d-workload-variant-autoscaler/pull/900). | ||
| 5. **Maintain framework-agnostic interface**: the FMA benchmark lifecycle (deploy, measure, | ||
| teardown) should remain pluggable into other benchmarking frameworks beyond llm-d-benchmark. | ||
|
|
||
|
|
||
| ## Legacy Benchmark Tooling | ||
|
|
||
| See [benchmark_legacy.md](benchmark_legacy.md) for documentation on the original `benchmark_base.py` tool. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.