[NVBUG-6448152][test] measure local-quiescence reclamation

[chienchunhung]

Diagnostic only — do not merge. This is the second arm of the NVBUG#6448152 historical-tree experiment. It keeps #16580's exact asynchronous global-agreement traffic and adds only rollback-free, local-success KV reclamation before global commit.

Question

After removing the synchronous PP barrier from the scheduler path, does retaining CTX KV until global commit still limit throughput?

Design delta from #16580

• C++ surfaces a local-success notification only after the local NIXL sender future is terminal and its sender session is released.
• Python frees that rank's primary KV allocation only when local success genuinely precedes the global commit.
• The request remains tracked in DISAGG_CONTEXT_TRANS_IN_PROGRESS; final state, error publication, response publication, and removal from global transfer bookkeeping still require the coordinator's global commit.
• Local failure never triggers early reclamation. A same-poll local success plus global commit uses the normal global-finalization path and is not counted as early reclaim.
• The existing two-element check_context_transfer_status contract remains unchanged; the diagnostic uses a separate opt-in status method.

Fail-closed scope

The experiment activates only for the exact qualified shape: C++ transceiver + asynchronous coordinator, NIXL, TP1/CP1/PP>1, attention DP off, primary V1 non-Mamba/non-VSWA KV manager, CTX max batch size 1, block reuse off, no connector, no separate draft KV manager, and exactly one transfer producer. The request must still be in the transfer-in-progress state.

Early release temporarily creates PP-local capacity skew. Rank 0 remains the sole scheduling authority and propagates one schedule; follower stages already gate execution with can_schedule() and poll for local transfer completion. This prevents different batches from executing, but may produce pipeline bubbles or retry exhaustion. Any such stall is a safety result, not a usable throughput result. A production design should make the propagated schedule depend on an explicit request/epoch reclaim credit rather than polling for any completion.

Exact experiment

• Parent experiment commit: dc06e82a182671a0d9626243d660a44a3b73bd1d ([NVBUG-6448152][test] isolate PP rendezvous from global commit #16580)
• Local-reclaim experiment commit: c108fde7a37094430256c2f449ee453e3e519210
• Experiment tree: 3b3256d1d846b4521ddc99b12ec5136c4784113d
• CI head: 5a4e3b29b7a7d0120a9533d18390c4b71817c655
• No-tree-change CI merge parent: 176041ba15f25fda876ccfa209d04896a003dc68
• [https://nvbugs/6342844][fix] Prevent disaggregated KV transfer stalls #15737 dataTransceiver.cpp blob preserved: b3eb1c0f984394e798993708bebcabe885ea1644
• Activation adds TRTLLM_NVBUG_6448152_CTX_LOCAL_KV_RECLAIM=1 while retaining TRTLLM_NVBUG_6448152_CTX_COORDINATOR_CONSENSUS=1
• Marker: NVBUG6448152_RECLAIM; early-release checkpoints are bounded to counts 1/128/256/384/512 per rank. The coordinator/last PP rank may legitimately have near-zero pre-commit releases.

Targeted validation

Stage: GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1

Selector: disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL

Validity requires 512/512 successful requests, no PP retry exhaustion, expected coordinator markers, substantial pre-commit early releases on early PP stages, and normal eventual global cleanup on every rank. Any failed request censors throughput.

Dependency and experiment graph

flowchart LR
    C["#16565<br/>pre-#15139 + #15737<br/>1557.83 tok/s"]
    T["#16566 / #16567<br/>post-#15139 + #15737<br/>~818 tok/s before teardown"]
    A["#16580<br/>async PP agreement<br/>global KV retention"]
    B["#16581<br/>same agreement traffic<br/>local-success KV reclamation"]
    L["#16572<br/>local-only unsafe control<br/>1564.71 tok/s"]
    C -->|"#15139 only"| T
    T --> A
    A --> B
    T --> L

Loading

Interpretation: #16580 at least 90% of #16565 means exposed rendezvous dominated and this arm should be similar. #16580 low but this arm at least 90% means retained KV/admission was the remaining dominant cost. Partial recovery means both contribute. Both low means another #15139 mechanism or a protocol implementation issue remains.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #60201 [ run ] triggered by Bot. Commit: 5a4e3b2 Link to invocation

[tensorrt-cicd]

PR_Github #60201 [ run ] completed with state FAILURE. Commit: 5a4e3b2
/LLM/main/L0_MergeRequest_PR pipeline #48571 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #60209 [ run ] triggered by Bot. Commit: a31c86b Link to invocation

[tensorrt-cicd]

PR_Github #60209 [ run ] completed with state FAILURE. Commit: a31c86b
/LLM/main/L0_MergeRequest_PR pipeline #48586 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #60255 [ run ] triggered by Bot. Commit: 993206e Link to invocation

[tensorrt-cicd]

PR_Github #60255 [ run ] completed with state SUCCESS. Commit: 993206e
/LLM/main/L0_MergeRequest_PR pipeline #48616 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #60387 [ run ] triggered by Bot. Commit: c86d111 Link to invocation

[tensorrt-cicd]

PR_Github #60387 [ run ] completed with state FAILURE. Commit: c86d111
/LLM/main/L0_MergeRequest_PR pipeline #48725 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation