[RFC] RL CI Matrix for vLLM: Behavioral + Physical + Protocol Coverage

[aoshen02]

Background

vLLM is the dominant rollout engine for post-training frameworks (verl, slime, OpenRLHF, SkyRL, AReaL, prime-rl, NeMo-RL, ROLL, miles, trl, open-instruct). The RL lifecycle exercises a unique code surface: cumem-based GPU memory management (sleep/wake_up), scheduler gating (pause/resume), and a weight-transfer HTTP protocol (/start_weight_update → /update_weights → /finish_weight_update).

This surface has seen a burst of production bugs in recent months — six currently open as PRs (#45326, #45552, #45363, #45518, #44972, #44483) — yet has zero dedicated CI coverage today.

This RFC proposes a structured, 5-tier RL CI matrix covering the production HTTP endpoint surface, grounded in a comprehensive survey of how 12 downstream frameworks already test this surface.

---

Problem: Current State

vLLM

• tests/basic_correctness/test_cumem.py covers the internal Python allocator API only — not the HTTP endpoints that RL frameworks call.
• tests/entrypoints/serve/dev/test_sleep.py asserts flag values (is_sleeping, Prometheus metrics) but never asserts behavioral consequences: it never checks whether the scheduler actually blocks requests or whether FlashAttention runs on unmapped KV cache.
• No test exists for /pause, /resume, /is_paused, /start_weight_update, /update_weights, /finish_weight_update, /get_world_size, PauseMode semantics (abort/wait/keep), or the compound RL training-step sequence.

Active bugs this CI would catch

PR / Issue	Bug	Status (updated)
#45552 open	sleep() / wake_up() missing torch.cuda.synchronize() → "200 lie": HTTP 200 returned while engine is already dead.	🟡 OPEN; covered by sleep/wake cycle tests
#45326 open	Request arriving during sleep() bypasses scheduler pause → hangs forever.	🟡 OPEN; CI test: test_sleep_abort_mode_blocks_new_requests
#45518 open	sleep() on already-sleeping engine returns no idempotency signal in response body.	🟡 OPEN; CI test: test_double_sleep_idempotent
#44972 open	Pages remapped by wake_up() may contain stale data → Mamba/GDN models produce NaN logprobs.	🟡 OPEN; ❌ needs Tier 2 model-specific test
#46438 open	discard(tag) → sleep() → wake_up() double unmap_and_release → CUDA_ERROR_INVALID_VALUE. Found by @AlanFokCo in multi-model hot-swap.	✅ test_cumem_physical.py

---

Directory Layout

Tests are organized by roadmap section. Each file covers both single-GPU and multi-GPU (TP≥2) cases via parametrize.

tests/entrypoints/serve/dev/rlhf/
├── conftest.py                              ← Shared fixtures (server harness, HTTP helpers, GPU memory query)
│
├── state_transitions/                       ← Section 2: State Transitions
│   ├── test_sleep_wake.py                   ← 2.3 Scheduler: sleep/wake/pause/resume lifecycle
│   ├── test_state_machine.py                ← 2.3 Scheduler: CPU state-machine invariants
│   ├── test_state_machine_enforcement.py    ← 2.3 Scheduler: transition enforcement
│   ├── test_cumem_physical.py               ← 2.3 Scheduler: CuMemAllocator VMM edge cases (@AlanFokCo)
│   ├── test_weight_update.py                ← 2.1 Weight Update: transfer protocol + model matrix (dense/MoE/FP8/VLM)
│   ├── test_weight_checker.py               ← 2.1 Weight Update: SHA-256 checksum verification
│   └── test_weight_update_metrics.py        ← 2.1 Weight Update: Prometheus metrics
│
├── info_retrieval/                          ← Section 3: RL Information Retrieval
│   ├── test_weight_info.py                  ← 3.1 Versioning: /weight_info, /update_weight_label
│   └── test_response_schema.py              ← 3.2 APIs: structured JSON responses for /sleep, /wake_up
│
└── consistency/                             ← Section 4: Consistency
    ├── test_routed_experts.py               ← 4.2 R3: routed_experts capture + replay correctness
    # 4.1 Logprobs: tests live in tests/v1/sample/test_logprobs.py (upstream)
    #              → see tests/v1/sample/test_prompt_logprobs_mode.py (#47680)
    #              → RL-specific: logprob stability across sleep/wake added in test_sleep_wake.py

Section → file mapping

Roadmap Section	Test File(s)	Upstream tests (not in this dir)
2.1 Weight Update	state_transitions/test_weight_update.py, test_weight_checker.py, test_weight_update_metrics.py (includes KV prefix cache flush after weight update)	—
2.2 Scheduler	state_transitions/test_sleep_wake.py, test_state_machine.py, test_state_machine_enforcement.py, test_cumem_physical.py (includes KV invalidation after wake, FP8 KV scale restoration, KV CPU backup)	—
3.1 Versioning	info_retrieval/test_weight_info.py	—
3.2 APIs	info_retrieval/test_response_schema.py	—
4.1 Logprobs	RL-specific cases in state_transitions/test_sleep_wake.py (logprob stability across lifecycle)	tests/v1/sample/test_logprobs.py, test_prompt_logprobs_mode.py (#47680)
4.2 R3	consistency/test_routed_experts.py	—
4.3 DSA Index	(not yet implemented)	—

Cleanup: scattered RL tests to consolidate

RL-related tests currently scattered across the repo. These should be moved into tests/entrypoints/serve/dev/rlhf/ when splitting PR #45586.

Current location	Content	Move to
tests/entrypoints/serve/dev/test_sleep.py	Sleep flag + Prometheus metrics (flag-only, no behavioral assertions)	state_transitions/ — superseded by test_sleep_wake.py
tests/v1/worker/test_sleep_mode_backend.py	Sleep mode backend unit test	state_transitions/
tests/basic_correctness/test_mem.py	CuMemAllocator internal Python API	state_transitions/ — colocate with test_cumem_physical.py
tests/entrypoints/weight_transfer/test_weight_transfer_llm.py	Weight transfer LLM E2E	state_transitions/ — same scope as test_weight_update.py

Open PRs — should land directly into rlhf/ instead of ad-hoc locations:

PR	Proposed file	Move to
#44972	tests/models/language/generation/test_gdn_sleep_wake.py	state_transitions/test_sleep_wake.py as Mamba/GDN parametrize case

Upstream tests to keep in place (reference via soft link in directory tree):

Location	Reason to keep
tests/v1/sample/test_logprobs.py	General logprobs, not RL-specific
tests/v1/determinism/test_batch_invariance.py	General determinism infrastructure

conftest.py — shared infrastructure

Component	Reference Pattern
ServerContext (subprocess launch/teardown)	miles with_mock_server() context manager
wait_for_health(url, timeout)	sglang wait_for_server
get_gpu_free_bytes()	sglang _platform_mem_usage
get_prometheus_metrics(url)	sglang weight_checker
generate_completion(url, prompt, ...)	common across all

---

Test Matrix with Downstream Mapping

state_transitions/test_state_machine.py (CPU, no GPU)

Industry-wide gap: No framework — including PR #45586 — tests sleep/wake state-machine invariants at the CPU level.

Test Scenario	Status	Reference Framework	Reference File	Reference Functions
sleeping_tags set ops (partial/full wake)	🆕 new	sglang	test/registered/rl/test_pause_generation_tensor_consistency.py	test_all_finished_extend_does_not_corrupt_running_batch, test_partial_finished_trims_correctly, test_empty_running_batch_stays_empty, test_bug_reproduction_without_fix
State machine thread safety (concurrent calls)	🆕 new	AReaL	tests/test_staleness_manager.py	TestThreadSafety (ThreadPoolExecutor concurrent)
CPU-only import isolation (stub GPU deps)	🆕 new	verl	tests/workers/rollout/test_vllm_weight_update_utils_on_cpu.py	test_split_buffer_updates_routes_registered_buffers, test_vllm_update_weights_loads_params_and_buffers
PauseMode Literal validation	🆕 new	—	—	—
is_sleeping = is_scheduler_paused OR executor.is_sleeping	🆕 new	—	—	—
CuMemAllocator.is_resident bookkeeping	🆕 new	—	—	—

state_transitions/test_sleep_wake.py — Physical Memory

Test Scenario	Status	Reference Framework	Reference File	Reference Functions
Level 1/2 GPU memory release	✅ done	sglang	test/registered/rl/test_release_memory_occupation.py	test_release_and_resume_occupation (tp=1,2)
Staged release (kv_cache → weights)	✅ done	sglang	test_release_memory_occupation.py	test_multi_stage_release_and_resume
Level 0 no memory change	✅ done	—	—	—
Prometheus cross-validation	✅ done	—	—	—
Wake recovers memory	✅ done	sglang	test_multi_instance_release_memory_occupation.py	_run_sglang_subprocess
Memory leak cycle (10+ rounds)	🆕 add	ROLL	tests/third_party/vllm/test_vllm_mem_oom.py	generate_memory (20 iter, load_format="dummy", CPU RSS tracking)
CPU weight backup release/resume	🆕 add	sglang	test_release_memory_occupation.py	test_release_and_resume_occupation_with_weights_cpu_backup

state_transitions/test_sleep_wake.py — Scheduler Gate

Test Scenario	Status	Reference Framework	Reference File	Reference Functions
Partial-wake blocks until KV resident (#44483)	✅ done	sglang/ROLL	test_release_memory_occupation.py / test_vllm_local.py	test_release_and_resume_occupation / test_vllm_with_load_offload
Sleep abort blocks new requests (#45326)	✅ done	ROLL	tests/third_party/vllm/test_abort.py	_check_vllm_abort
Level 0 blocks without memory change	✅ done	—	—	—
PauseMode abort/wait/keep	✅ done	SkyRL	gpu/gpu_ci/test_pause_and_continue_generation.py	test_pause_generation, test_continue_generation, test_pause_continue_cycle
Double sleep/wake idempotent (#45518)	✅ done	—	—	—
Abort then sleep/wake	✅ done	—	—	—
Concurrent sleep+generate race (#45326)	🆕 add	miles	tests/fast/router/test_session_race_conditions.py	TestSessionConcurrencyContracts (4 tests), TestClosingRaceConditions (5 tests) — 8-thread ThreadPoolExecutor
sampling_n > 1 + abort interaction	🆕 add	ROLL	tests/third_party/vllm/test_abort.py	_check_vllm_sampling_n — cross-version (CancelledError vs finish_reason)
Sleep/wake latency assertion	🆕 add	sglang	test_update_weights_from_distributed.py	assert time < 3s

state_transitions/test_sleep_wake.py — Output Correctness

Test Scenario	Status	Reference Framework	Reference File	Reference Functions
Golden output roundtrip (level 1)	✅ done	sglang	test_update_weights_from_disk.py	TestEngine/ServerUpdateWeightsFromDisk
Staged wake output matches	✅ done	—	—	—
3-cycle stability	✅ done	—	—	—
Prefix cache flush after wake	✅ done	—	—	—
Prefix cache flush after weight update	✅ done	sglang	test_weight_checker_e2e.py	test_e_checksum_*
Logprobs precision pre/post sleep	🆕 add	ROLL	tests/distributed/strategy/log_probs/ (9 files)	test_fsdp_log_probs_full/lora/cp/cp_rmpad, etc.
Logprobs precision match (server vs in-process)	🆕 add	trl	tests/test_vllm_client_server.py	test_logprobs_match_with_non_default_sampling (xfail)

state_transitions/test_weight_update.py — Weight Transfer Protocol

Test Scenario	Status	Reference Framework	Reference File	Reference Functions
Start/finish framing (no tensors)	✅ done	AReaL	tests/experimental/weight_update/test_wu_controller.py	TestLifecycle (3 tests)
Finish without start	✅ done	—	—	—
Prefix cache flush after finish	✅ done	sglang	test_weight_checker_e2e.py	(checksum compare)
Real tensor push → output changes	✅ done (skip)	sglang	test_update_weights_from_tensor.py	TestUpdateWeightsFromTensor::test_direct
Full RL step compound	✅ done	—	—	—
2× repeated RL steps stable	✅ done	—	—	—
MoE × quant reload matrix	✅ done	—	—	—
get_world_size positive	✅ done	trl	test_vllm_client_server.py	TestVLLMClientServerTP
Weight checksum (SHA-256 snapshot/compare)	🆕 add	sglang	test/registered/rl/test_weight_checker_e2e.py	test_a_snapshot_then_compare_unchanged_succeeds, test_c_update_with_diff_tensor_makes_compare_fail, test_e_checksum_stable, test_e_checksum_changes_after_update, test_e_checksum_skips_non_persistent_buffers
Protocol error paths (dtype rejection)	🆕 add	ROLL	tests/third_party/vllm/test_collective_rpc.py	test_vllm_collective_rpc (CUDA tensor ❌, CPU tensor ❌, numpy ❌, Python list ✅)
Start repeated / update timeout	🆕 add	AReaL	tests/experimental/weight_update/test_wu_controller.py	TestConnect (3), TestUpdateWeights (4), TestDisconnect (3)
Weight update latency assertion	🆕 add	sglang	test_update_weights_from_distributed.py	assert time < 3s

Model-specific and multi-GPU coverage

Model-specific and multi-GPU tests are parametrized within each file (not separate files). Strategy: 1-GPU smoke with tiny-random models; TP≥2 / nightly models marked @pytest.mark.nightly.

state_transitions/test_sleep_wake.py — model × parallelism matrix

Test Scenario	Reference	GPU
MoE sleep/wake + generate	sglang test_moe_model_release_and_resume (tp=2, ep=2)	2-4
Mamba/GDN hybrid sleep/wake (#44972)	sglang test_hybrid_mamba_model_release_and_resume (tp=4)	4
VLM sleep/wake	ROLL test_fsdp2_cp_vlm_*	2+
TP=2 staged sleep/wake + per-GPU memory	sglang test_multi_instance_release_memory_occupation.py	4

Gaps:

• MoE sleep/wake smoke test (SkyRL#1680 reproducer)
• Mamba/GDN hybrid sleep/wake test ([Test][V1] Add sleep/wake correctness regression test for hybrid GDN/… #44972 reproducer)
• VLM sleep/wake + weight-reload test
• ≥3 model architectures covered: dense, MoE, VLM

state_transitions/test_weight_update.py — model × parallelism matrix

Test Scenario	Reference	GPU
FP8 weight reload	ROLL test_fp8 (Fp8Worker custom_wakeup)	TP=2
Blackwell FP8/FP4 weight update	sglang test_update_weights_from_disk_blackwell.py	4×B200
FP32 LM head precision	sglang TestLMHeadFP32	1
VLM weight update	trl TestVLLMClientServerVLM	1
NCCL distributed weight update	sglang test_update_weights_from_distributed.py	2+
TP=2 generate + weight update	trl TestVLLMClientServerTP	3
DP=2 generate + weight update	trl TestVLLMClientServerDP	3
All parallel combos (DP/TP/PP/CP/EP)	AReaL test_nccl_integration.py (15 tests)	2-8

Gaps:

• MoE weight-reload smoke test (SkyRL#1680 reproducer)
• MoE + FP8 weight reload test (#45835 reproducer)
• Multi-backend matrix: FlashInfer / Triton / CUTLASS for weight reload

state_transitions/test_cumem_physical.py — CuMemAllocator VMM edge cases (@AlanFokCo)

Blocked on #46438 (CuMemAllocator.discard()).

• discard(tag) → sleep() → wake_up() double-unmap guard
• sleep(keep_tags=("kv_cache",)) → wake_up() already-mapped skip
• CUDA graph pool VMM slowdown regression (3-5x with graph pool)
• Cross-tag wake ordering ([Bug]: wake_up(tags=["weights"]) + forward → illegal memory access (KV cache still asleep) #44395 reproducer)
• Ensure [Core] Add CuMemAllocator.discard() for tag-selective GPU memory release #46438 merged — prerequisite
• NCCL memory reclamation: ~958 MiB/GPU freed during sleep on TP>1 (#46234, blocked on NCCL 2.29.7)

consistency/test_routed_experts.py — R3 replay correctness

Test Scenario	Reference	GPU
MoE routed_experts capture	sglang TestReturnRoutedExperts	4-8
CUDA IPC device remapping	sglang test_monkey_patch_torch_reductions	2

Gaps:

• R3 routing-replay correctness after weight update (SkyRL#1846 P0)
• FlashInfer backend coverage (#44116 all-zero bug)

---

Work Estimate

File	New Tests	Complexity	GPU
conftest.py	— (refactor)	Low	—
state_transitions/test_state_machine.py	~8	Low (CPU)	None
state_transitions/test_sleep_wake.py additions	~6 + model matrix	Medium	1-4 GPU
state_transitions/test_weight_update.py additions	~4 + model matrix	Medium	1-4 GPU
consistency/test_routed_experts.py	~4	Medium	2-8 GPU
Total	~25 new

---

Files Already Written (PR #45586)

File	Tests	Tier Coverage
tests/entrypoints/serve/dev/rlhf/test_sleep_wake.py	26	Tier 1A, 1B, 1C
tests/entrypoints/serve/dev/rlhf/test_weight_update.py	11	Tier 1D

All tests drive the server HTTP endpoint surface (VLLM_SERVER_DEV_MODE=1), require --enable-sleep-mode, and are self-contained.

---

---

Implementation Progress

Phase 0 — Test Infrastructure ✅ (complete)

File	Tests	Status	Commit
conftest.py	shared fixture	✅ done	e9e3b8e
test_state_machine.py	16	✅ done	4cfde2e
test_sleep_wake.py additions	28 total	✅ done	f542fcf
test_weight_update.py additions	16 total	✅ done	f7c24d2
test_multi_gpu.py (TP=2)	4	✅ done	615a1ca

Phase 1 — Structured JSON responses ✅ (complete)

POST /sleep and POST /wake_up now return structured JSON instead of bare Response(200):

// /sleep → {"status": "sleeping", "level": int, "elapsed_ms": float}
// /wake_up → {"status": "awake"|"sleeping", "tags_woken": list|None, "elapsed_ms": float}

Removes 2 stale FIXME comments about v0 frontend multiprocessing (verified: not a real bug in v1 engine).

File	Tests	Status	Commit
vllm/entrypoints/serve/dev/sleep/api_router.py	—	✅ done	7e6abd2
test_response_schema.py (new, 5 tests)	5	✅ done	7e6abd2

Phase 2 — /weight_info + /update_weight_label ✅ (complete)

New weight version endpoints tracking weight generation and label:

// GET /weight_info → {"weight_gen": int, "weight_label": str}
// POST /update_weight_label → {"weight_label": "step-1500"}

Design improvements over sglang: weight_gen is an internal monotonic counter (auto-incremented by finish_weight_update, can't go backwards), decoupled from the human-readable weight_label.

File	Tests	Status	Commit
vllm/entrypoints/serve/dev/rlhf/api_router.py	—	✅ done	3a368c3
test_weight_info.py (new, 8 tests)	8	✅ done	3a368c3

E2E Validation in inferactinc/public:vime-latest ✅

Full test suite validated in the vime RL container (aarch64, Python 3.12, vLLM 0.23.0, GB200 NVL):

File	Result	Notes
test_state_machine.py	16/16 ✅	8m42s
test_response_schema.py	5/5 ✅	3m05s (Phase 1 patch)
test_weight_info.py	8/8 ✅	4m35s (Phase 2 patch)
test_sleep_wake.py	28/28 ✅	16m23s
test_weight_update.py	11/11 ✅, 5 skip	6m12s (MoE models not pre-cached)
test_multi_gpu.py	4/4 ✅	4m01s (TP=2 on GB200)
Total	72 passed, 5 skipped

Bugs fixed during vime e2e run: (1) re_allocated_gib sign was inverted (wake re-allocates, free bytes decrease); (2) is_sleeping after /pause is True in v1 engine (composite predicate); (3) errno.EROFS check instead of locale-sensitive string matching.

[AlanFokCo]

@aoshen02 This is a great framework — the 5-tier structure (flag → behavioral → physical → protocol → E2E) is exactly the right decomposition.

A few edge cases from our cross-architecture model switching work that could extend the matrix, particularly at Tier 2 (Physical) and Tier 3 (Protocol):

Physical (cumem state machine):

• discard(tag) → sleep() → wake_up(): the discarded allocation has is_asleep=True before sleep; sleep must skip it to avoid double unmap_and_release → CUDA_ERROR_INVALID_VALUE. We hit this in production and it's now guarded in [Core] Add CuMemAllocator.discard() for tag-selective GPU memory release #46438.
• sleep(keep_tags=("kv_cache",)) → wake_up(): when some allocations stay mapped during sleep, wake_up() must skip them (they're not asleep). Without the is_asleep guard, create_and_map on an already-mapped VA crashes.
• Graph pool VMM slowdown: sleep()/wake_up() take 3-5x longer when a CUDA graph capture pool is present. Test that timing doesn't regress by >Nx when enforce_eager=False.

Protocol (cross-tag ordering):

• wake_up(["weights"]) → forward → wake_up(["kv_cache"]): if a forward dispatches between the two wakes, it touches released KV memory ([Bug]: wake_up(tags=["weights"]) + forward → illegal memory access (KV cache still asleep) #44395). Assert clean error, not segfault.
• wake_up(["kv_cache"]) → wake_up(["weights"]): waking KV before weights means flash_attn may access zeroed weight memory if a forward accidentally runs. Assert that scheduler remains paused until full residency.

Happy to contribute test implementations for these cases.

[aoshen02]

@aoshen02 This is a great framework — the 5-tier structure (flag → behavioral → physical → protocol → E2E) is exactly the right decomposition.

A few edge cases from our cross-architecture model switching work that could extend the matrix, particularly at Tier 2 (Physical) and Tier 3 (Protocol):

Physical (cumem state machine):

• discard(tag) → sleep() → wake_up(): the discarded allocation has is_asleep=True before sleep; sleep must skip it to avoid double unmap_and_release → CUDA_ERROR_INVALID_VALUE. We hit this in production and it's now guarded in [Core] Add CuMemAllocator.discard() for tag-selective GPU memory release #46438.
• sleep(keep_tags=("kv_cache",)) → wake_up(): when some allocations stay mapped during sleep, wake_up() must skip them (they're not asleep). Without the is_asleep guard, create_and_map on an already-mapped VA crashes.
• Graph pool VMM slowdown: sleep()/wake_up() take 3-5x longer when a CUDA graph capture pool is present. Test that timing doesn't regress by >Nx when enforce_eager=False.

Protocol (cross-tag ordering):

• wake_up(["weights"]) → forward → wake_up(["kv_cache"]): if a forward dispatches between the two wakes, it touches released KV memory ([Bug]: wake_up(tags=["weights"]) + forward → illegal memory access (KV cache still asleep) #44395). Assert clean error, not segfault.
• wake_up(["kv_cache"]) → wake_up(["weights"]): waking KV before weights means flash_attn may access zeroed weight memory if a forward accidentally runs. Assert that scheduler remains paused until full residency.

Happy to contribute test implementations for these cases.

Let me add you to my branch, so that we can cooperate.

[AlanFokCo]

Sounds great! My handle is @AlanFokCo.

I can start with the discard → sleep → wake_up state machine tests (Tier 2) — already have fixtures from #46438 that fit your layout. Also have a reliable H20 repro for the wake_up(["weights"]) → forward IMA (Tier 3).

Let me know once I have access.