Design: safe cross-process admission for concurrent physical KV growth

[shipiyouniao]

Summary

Concurrent kvcached instances on the same physical GPU need cross-process admission for physical KV growth.

#418 makes VMM allocation and mapping failures transactional, and #453 converts physical-capacity exhaustion into a scheduling miss. Those changes are necessary, but they do not arbitrate capacity between processes. Under sustained colocated pressure, both instances can observe the same free-memory snapshot, admit overlapping growth, and race in cuMemCreate().

This is no longer only a code-review concern. We can reproduce the race reliably with two TP=4 serving instances under high concurrency. Without cross-process admission, the CUDA driver rejects physical-page allocations with OOM. Transactional rollback prevents a fatal process abort, but repeated OOM attempts become allocation misses, request failures, and severe asymmetry.

A similar race was discussed as a possibility in #221, but that report stopped reproducing after its TP socket collision was fixed. The reproduction here uses separated instance IPC and current transaction handling, so it is a distinct remaining problem.

Evidence

Transactional rollback without cross-process admission

Two high-pressure rounds were run against two colocated instances:

Round	Instance A	Instance B
1	411 succeeded, 0 failed	55 succeeded, 356 failed
2	115 succeeded, 296 failed	411 succeeded, 0 failed

The two instances logged 1,932 and 2,552 raw CUDA physical-page OOM failures respectively. There was no process restart or fatal CUDA error because the failures were rolled back, but request-level correctness and fairness were not acceptable.

This is the gap between rollback and admission:

1. both processes read sufficient free memory;
2. both start physical growth;
3. the combined request exceeds real GPU capacity;
4. the CUDA driver rejects the later cuMemCreate();
5. fix: make KV tensor VMM batches transactional #418 rolls back the partial transaction;
6. fix(vllm): report an exhausted KV pool as a scheduling miss, not a crash #453 turns exhaustion into an allocation miss;
7. sustained retries can still starve one instance and fail requests.

Admission designs evaluated

1. Full physical-growth transaction lock (#455)

The original #455 design:

• keys a process-shared file lock by physical GPU;
• refreshes mem_get_info while holding the lock;
• checks headroom against the current snapshot;
• holds the lock across the complete physical-growth mapping transaction.

Its focused reproducer changes the uncontrolled race into one successful growth and one recoverable capacity_exhausted result, with both processes exiting normally.

Among the designs tested so far, this remains the strongest correctness boundary and the best current candidate. It makes the capacity decision and the corresponding physical growth one transaction. However, its cost concern is valid: file-lock acquisition, device selection, memory query, and a relatively broad critical section are added to successful growth.

2. Smaller per-page transactions and fairness retries

A version that reduced growth to one page per transaction and retried misses removed raw CUDA OOM and completed all requests, but produced stable starvation:

Metric	Baseline	Candidate
Aggregate effective throughput	3,246 tok/s	1,574-1,600 tok/s
Change	-	-50.7% to -51.5%
Slow-side median TTFT	about 3 s	about 70 s

Mutual exclusion alone did not provide fair admission. One process repeatedly reacquired capacity while the other accumulated retries.

3. Worker-owned all-TP headroom admission

We also evaluated worker-owned admission where each physical growth queried all TP workers and rejected the batch if any rank lacked headroom. It avoided OOM and permanent hangs, and both instances completed 390/390 requests, but it moved a distributed control round trip into the allocation hot path:

Metric	Change
Elapsed time	+40%
Output throughput	-28.5%
Mean TTFT	+63%
Headroom rejections	2,182 / 2,176
Allocation misses	1,102 / 1,096

Removing that hot-path admission restored average throughput to within about 1.4% in the same workload, which isolates the regression to the admission path rather than VMM rollback itself. More detail is available in #416.

4. No admission, rely only on the CUDA driver and rollback

This has the lowest uncontended overhead, but the first result above shows that it is not sufficient under sustained colocated pressure. The CUDA driver is a final atomic gate, not a fair or efficient cross-process allocator.

Current conclusion

#418 and #453 remain required, but they do not replace cross-process capacity admission.

All alternatives tested after the original #455 shape either:

• allow repeated physical allocation OOM and request failures;
• introduce a distributed slow path on every growth attempt; or
• avoid OOM at the cost of severe starvation and roughly 50% aggregate throughput loss.

For that reason, #455 has been reopened as a draft. Its original transaction boundary is currently the best candidate among the tested designs, but it should not merge until its uncontended cost and fairness under sustained contention are addressed.

A promising refinement is to keep a process-shared transaction boundary while shortening the critical section around physical capacity reservation, rather than holding it across Python IPC, TP response waits, or synchronization. That needs an explicit ownership model for reserved bytes or handles so the lock cannot be released before the capacity decision becomes real.

Design questions

1. Can physical handles be reserved under a short process-shared lock, then mapped outside the lock without reopening the TOCTOU race?
2. Should admission use bounded chunks or tickets to prevent one instance from repeatedly acquiring all newly available capacity?
3. What is the right cross-container coordination primitive and deployment contract?
4. How should admission expose contention, wait time, failed reservations, and fairness without placing worker RPC in the allocation hot path?
5. Can the uncontended path remain close to the current rollback-only baseline while preserving a coherent capacity snapshot?

Acceptance criteria

A replacement for the current draft should demonstrate all of the following with at least two colocated instances:

• no raw physical-page OOM during expected capacity contention;
• no process abort, illegal memory access, or partial VMM state;
• 100% valid request completion unless explicitly cancelled;
• no instance remains without progress while another repeatedly grows;
• bounded admission wait and retry counts;
• no material aggregate throughput or TTFT regression in the uncontended case;
• a documented shared coordination contract for multi-container deployments.

[shipiyouniao]

Update from the latest FIFO admission validation:

The rank-local FIFO admission variant should be rejected. This is a functional regression, not just a performance concern.

In a two-instance high-concurrency run, each instance received 390 requests:

Instance	Succeeded	Failed
A	202	188
B	196	194

Both model processes restarted twice and did not remain ready. The observed failure chain was:

waiting for fair physical growth turn
-> physical GPU headroom would be crossed
-> physical page allocation failed
-> allocation miss
-> TP collective / CUDA event failure
-> EngineCore fatal restart

The underlying flaw is that each TP worker participates in FIFO admission independently. A single distributed map can therefore have some ranks complete physical growth while another rank is rejected. Returning capacity_exhausted from the middle of that collective leaves the TP operation split before compensating rollback can restore a safe execution boundary.

This result does not remove the need for cross-process admission. It means the admission unit must be the complete TP/PP map transaction:

1. resolve the full physical GPU set for the transaction;
2. acquire one fair turn for that complete set before sending any worker map request;
3. broadcast the worker maps concurrently;
4. collect every response;
5. if any rank fails, issue compensating unmap to every rank before releasing the turn.

A revised prototype using that boundary has passed focused Linux/CUDA tests and a two-process T4 VMM contention probe. Across 20 contention rounds, both processes exited normally and made exactly equal progress (160 mapped pages each, with 20 recoverable headroom rejections each). Its uncontended real-map mean was within measurement noise of the no-admission path. Full serving validation is still required before proposing it for merge.

[shipiyouniao]

Correction to the prototype result above: the first control-side admission prototype is not valid for merge.

A TP=4 serving validation exposed a CUDA thread-local device bug in the worker listener. The listener queried current_device_pci_bus_id() from its own background thread. CUDA current-device state is thread-local, and a new listener thread defaults to device 0. As a result, TP0 through TP3 all reported the same PCI Bus ID and every admission ticket contained only one physical GPU. The supposed group transaction therefore protected TP0 but left the other ranks outside the lock.

The first request round completed, but a delayed GPU fatal occurred afterward. The host reported Xid 13/31/43 and multi-GPU FAULT_PDE ACCESS_TYPE_VIRT_READ, followed by model and MPS restarts. This invalidates the earlier focused result as evidence for multi-GPU correctness.

The implementation has been corrected to capture the PCI Bus ID once on the initialized worker thread before starting the listener. The listener now returns that immutable value instead of querying CUDA thread-local state. A regression test explicitly makes the caller thread and listener thread report different devices and verifies that IPC returns only the caller-thread value.

The corrected version still requires a real multi-GPU TP serving validation. Until that passes, the revised prototype should be treated as experimental and should not be merged.

[shipiyouniao]

A second correction: the PCI Bus ID capture fix makes the ticket contain the complete TP GPU set, but the corrected prototype still fails real multi-GPU serving validation and must not be merged.

The first high-concurrency round completed successfully on both instances. About 59 seconds after the benchmark ended, MPS reported Server is handling a fatal GPU error. The host then recorded Xid 13/31/43 and FAULT_PDE ACCESS_TYPE_VIRT_READ across all four GPUs. The primary process, both secondary processes, and MPS subsequently restarted.

This means the failure was not explained solely by the listener thread reporting device 0. The complete-set lock serializes distributed map attempts, but it does not make the map itself atomic across TP ranks:

• each rank still performs its own headroom check and physical map;
• one rank can reject after another rank has already mapped;
• the controller then relies on compensating unmap across all ranks;
• that compensation is not equivalent to an all-rank prepare/commit protocol and has not prevented the delayed invalid-access failure.

There is also a concrete rollback hazard that now needs targeted evidence. FTensor::map() returns failure for an already-mapped page, while the controller's compensating rollback currently sends an ignore-missing unmap for the full offset set to every rank. If a rank enters this path with a pre-existing mapping, compensation can remove that existing mapping rather than only mappings created by the failed transaction.

The current prototype is therefore rejected. Further work should first capture transaction ID, group ID, page/offset, per-rank pre-map state, pages newly mapped by this transaction, and rollback results around the fatal run. A safe replacement needs either a real all-rank prepare/commit reservation or transaction-owned rollback that can only undo mappings created by that transaction. A group lock plus unconditional compensating unmap is not sufficient.

[shipiyouniao]

Another validation update: transaction-owned online rollback is still not safe enough for merge.

A no-injection baseline completed successfully on both instances: 195/195 requests each, 117,000 output tokens each, and zero restarts.

Under fail-before-map, only 374 and 397 output tokens were produced before MPS entered a fatal GPU state and all inference processes restarted. Under lose-success-response, both instances stopped at 7,427/117,000 output tokens; MPS failed about 35 seconds later, followed by restarts and CUDA error: invalid resource handle.

The implementation correctly avoided rolling back mappings that could be reconciled as prepared or committed. However, whenever any rank still had a confirmed map failure, it performed synchronous transaction-owned physical rollback. Real GPU validation shows that this remains unsafe: ownership prevents unmapping an older transaction's page, but it does not establish a safe CUDA execution boundary across asynchronous worker queues and other processes sharing MPS.

The next prototype therefore removes physical unmap from the distributed-map failure path. Successful rank mappings are retained as transaction-owned orphans and may only be adopted by a later retry for the same group and page offsets. If no retry occurs, they remain mapped until an explicit safe reclamation point or process teardown. This prototype is still experimental and will not be proposed for merge until it passes the same multi-GPU chaos cases.

[shipiyouniao]

Latest performance validation changes the conclusion again.

The orphan-retention/adoption path passed the multi-GPU chaos cases and is materially better than the earlier transaction-lock prototypes, which either starved one instance or lost roughly half of aggregate throughput. However, the current full-transaction FIFO admission is still not acceptable for merge.

Two co-located TP=4 serving instances completed 411/411 requests each with no restart, preemption, CUDA fatal, or request failure. Compared with the pre-admission baseline:

Metric	Baseline	Current prototype	Change
Aggregate output throughput	3250.08 tok/s	2308.54 tok/s	-28.97%
Mean TTFT, instance A	4272.52 ms	6957.46 ms	+62.84%
Mean TTFT, instance B	4305.75 ms	7899.53 ms	+83.46%
Mean TPOT, instance A	70.31 ms	90.40 ms	+28.57%
Mean TPOT, instance B	69.99 ms	113.69 ms	+62.44%
Instance throughput gap	0.23%	26.18%	worse

The slow side completed about 49 seconds later. It also observed more physical-capacity rejections and persistent waiting, while both sides remained functionally correct.

This confirms two separate results:

1. Retaining transaction-owned orphan mappings and adopting them on a matching retry is the strongest correctness result so far. It removes unsafe online physical rollback from the distributed-map failure path.
2. Holding the FIFO turn across worker IPC, CUDA mapping, synchronization, and transaction completion creates a slow and unfair critical section. It converts the safety problem into an admission/fairness bottleneck.

The next prototype will keep orphan retention, but replace the full map lock with a short process-shared reservation transaction: atomically reserve the requested physical bytes for the complete TP/PP device set, release the bookkeeping lock before worker IPC/CUDA mapping, then commit or retain the reservation with the resulting mapping ownership. Performance and fairness must return close to the pre-admission baseline before this is suitable for merge.

[shipiyouniao]

Follow-up performance result for the short native-lock prototype: correctness and fairness improved, but the implementation still does not meet the performance bar.

Metric	Baseline	Prototype	Change
Successful requests	411/411 per instance	411/411 per instance	unchanged
Aggregate output throughput	3250.08 tok/s	2467.12 tok/s	-24.09%
Mean TTFT	4272.52 / 4305.75 ms	7039.27 / 7129.56 ms	+64.8% / +65.6%
Mean TPOT	70.31 / 69.99 ms	95.86 / 95.77 ms	+36.3% / +36.8%
Inter-instance throughput gap	0.23%	0.013%	improved

Both instances remained healthy with zero restarts, OOMs, preemptions, CUDA fatal errors, or tracebacks. The remaining regression is therefore in the admission failure path rather than model execution or instance imbalance.

The trace showed repeated deterministic capacity_exhausted results. Each rejected distributed map still performed a transaction-state query and then broadcast orphan retention, so one known pre-map rejection could require up to three TP IPC rounds before becoming an allocation miss.

The next revision narrows this protocol:

• workers return transaction_state=not_prepared with a deterministic pre-map failure;
• only timeout, disconnect, or otherwise unknown outcomes trigger a state query;
• if every rank is not_prepared, the transaction fails immediately without orphan IPC;
• orphan retention is sent only to ranks that actually reached mapped or prepared.

This keeps the existing unknown-outcome and partial-success safety semantics while removing avoidable IPC from the common capacity-rejection path. The revised prototype still needs the same dual-instance performance and chaos validation before it can be considered ready.

[shipiyouniao]

The latest instrumentation narrows the remaining regression to the admission semantics rather than the lock implementation.

The metrics below are deltas across one dual-instance serving run. Maxima are cumulative maxima reported after the run.

Metric	Total	Per worker operation
Physical-growth transactions	144	-
Worker operations	576	-
Ticket wait	154,840 us	0.269 ms
Admission	341,223 us	0.592 ms
Admission excluding ticket wait	-	0.324 ms
Physical-handle reservation	177,997 us	0.309 ms
VMM map	1,176,489 us	2.043 ms

Every transaction contained exactly one logical offset. The run completed 822/822 requests with no restart, CUDA error, or traceback, but aggregate output throughput remained about 24.75% below the pre-admission baseline. It also produced 59 allocation misses and 472 capacity_exhausted records.

This rules out file-ticket scanning, lock acquisition, and the memory query as the primary source of the regression. Optimizing FIFO bookkeeping further is unlikely to recover the missing throughput.

The more important problem is that the current native ticket is acquired independently by each TP worker. One distributed TP/PP map is therefore still split into multiple rank-local capacity decisions. Two colocated instances can reserve physical handles on different ranks, then fail on another rank and retain the successful mappings as orphans. That is safe from online-unmap corruption, but it can fragment physical capacity across ranks and turn one logical page growth into repeated whole-transaction admission misses.

The next prototype changes the transaction boundary rather than the lock implementation:

1. Acquire one fair turn for the complete sorted physical-device set.
2. Broadcast a prepare phase: every rank creates transaction-owned physical handles with cuMemCreate, but does not modify the virtual mapping.
3. If any rank rejects preparation, destroy only the still-unmapped handles. No physical unmap is required.
4. If all ranks prepare successfully, release the group turn and broadcast commit so ranks map the prepared handles concurrently.
5. Preserve orphan retention/adoption only for an unknown or partially acknowledged commit result; never use online physical rollback.

The group turn covers only physical-handle preparation. Worker mapping, CUDA synchronization, commit acknowledgement, and finalization remain outside the critical section.

This is an implementation-in-progress design, not a validated result yet. It still needs the same dual-instance performance test and the fail-before-map / lost-success-response chaos cases before it can replace the current draft.

[shipiyouniao]

The full TP/PP prepare/commit prototype has now completed the same multi-GPU serving workload as the pre-admission baseline. The transaction and chaos cases pass, but the performance result rejects this revision for merge.

Both runs used 411 requests per instance at concurrency 137:

Metric	Pre-admission baseline	Prepare/commit prototype	Change
Successful / failed requests	822 / 0	822 / 0	unchanged
Aggregate output throughput	3250.08 tok/s	2677.58 tok/s	-17.61%
Inter-instance throughput gap	0.23%	0.16%	improved
Mean TTFT	4289 ms	14939 ms	+248.3%
Mean TPOT	70.15 ms	66.34 ms	-5.43%

The near-identical per-instance throughput and successful chaos validation show that group-wide preparation fixes the earlier rank-split and starvation behavior. However, the large TTFT increase and aggregate-throughput loss remain unacceptable.

The TPOT result is important: token generation itself is not slower. The regression is concentrated before generation, where physical-growth admission reduces the number of requests that can make progress concurrently and converts pressure into waiting/allocation misses.

This also means another round of file-ticket or lock micro-optimization is unlikely to recover the missing throughput. The next investigation should record, for every rejected growth attempt:

• physical free bytes before preparation;
• requested physical bytes for the complete page bundle;
• configured headroom and exact shortfall;
• logical blocks/tokens unlocked by that page bundle;
• time until the next successful growth or physical-capacity release.

The working hypothesis is now capacity quantization: one logical growth offset maps a complete page bundle across all relevant layers/buffers. Near the utilization boundary, rejecting one coarse bundle can remove substantial logical concurrency even though ticket, memory-query, reservation, and map latencies are individually small.

The prepare/commit ownership model remains the strongest correctness result so far, but this implementation is still experimental and should not merge until the capacity-bound throughput regression is resolved.