Skip to content

cleanup(rdna): delete env-gated debug probes and orphan modules - #45

Closed
BlivionIaG wants to merge 1 commit into
rdna_extrasfrom
cursor/delete-env-debug-probes-9929
Closed

BlivionIaG wants to merge 1 commit into
rdna_extrasfrom
cursor/delete-env-debug-probes-9929

Conversation

@BlivionIaG

@BlivionIaG BlivionIaG commented Oct 7, 2026 •

Copy link
Copy Markdown

Purpose

Subtraction-only cleanup of leftover owner forensics on rdna_extras. Env-unset behavior is unchanged: no new flags, helpers, state, or replacement logging, and vllm/envs.py is untouched.

Not a duplicate. Open PRs on opengfx1030/vllm-rdna (including #41 W4A8 trim, #40 hetero MoE, #37 expert residency, #33 FA decode, #31 W4A16 explore, #25 all-reduce) do not delete these probes. Hands-off files owned by those branches were not edited: vllm/envs.py, vllm/v1/worker/gpu/model_runner.py, vllm/v1/attention/backends/rdna_attn.py, vllm/models/qwen4_exp/amd/hyperconnection.py, vllm/model_executor/layers/quantization/exl3.py, vllm/distributed/device_communicators/rdna_all_reduce.py, csrc/, docs/, bench_results/.

AI assistance was used (Cursor cloud agent). No model-output, accuracy, or serving change: the deleted branches run only when an env var is set, and unset is the default.

Kept / dropped

Item Result
1. vllm/utils/rocm_poolwatch.py, vllm/utils/rocm_graph_keepalive.py Deleted. rg found no Python imports. Docs mention the keepalive filename only. C++ rdna2_immortal_zeros in csrc/ left in place.
2. GDN probes in qwen_gdn_linear_attn.py Deleted VLLM_LOG_GDN_PTRS, VLLM_GDN_OUT_DEBUG, VLLM_GDN_NAN_DEBUG, VLLM_GDN_STATE_DEBUG / VLLM_GDN_SAVE (torch.save to /tmp/gdn_l6.pt), VLLM_GDN_DBG. Kept VLLM_GDN_HIP_PREFILL and VLLM_GDN_DECODE_RDNA2.
3. VLLM_MAMBA_COPY_DEBUG and the debug local in mamba_utils.py Deleted. No test, script, or launcher reads the knob.
4. QSA _DEBUG / _qsa_dump_bad_rows Deleted the Python clone/print/dump. The HIP call qsa_compress_groups_rdna2 takes no debug argument, so the call is unchanged. Not dropped. csrc/rocm/qsa_rdna2.cu still getenvs VLLM_QSA_RDNA2_DEBUG for a printf; that file is hands-off.
5. CUDA graph probes Deleted VLLM_PIECE_IN_DEBUG, VLLM_CG_NAN_INPUT_CHECK, VLLM_CG_REPLAY_LOG. Kept VLLM_CG_SKIP_REPLAY and capture-time _nan_to_num_tree.
6. backends.py VLLM_CG_REPLAY_LOG and VLLM_PIECE_DUMP / VLLM_PIECE_TP Deleted.
7. VLLM_W4A16_PTR_DEBUG in _tp_all_reduce Deleted, including the function-local import os. Restored the pre-probe return get_tp_group().all_reduce(input_).
8. W4A16 dead AWQ prefill arm Deleted the VLLM_W4A16_PTR_DEBUG logs, the unreachable kernel_name == "awq_prefill" arm (selector returns only w4a8_prefill / prefill / exllama / rdna2_decode), the else-arm ops.awq_gemm_rdna2_prefill call, and unused _awq_prefill_available. Kept the gptq/exllama fallback. awq_gemm_rdna2_prefill is not in csrc/.
9. VLLM_BT_DEBUG in block_table.py and worker/utils.py Deleted those two blocks. Not removed from hands-off model_runner.py and rdna_attn.py.
10. VLLM_DEBUG_PREFIX_HIT Deleted. Kept VLLM_NO_PARTIAL_HASH and the existing VLLM_TRACE_PREFIX_CACHE logs (not in the delete list; they are a separate trace switch).
11. VLLM_MOE_NAN_DEBUG, VLLM_LAYER_NAN_DEBUG, VLLM_PLE_NAN_DEBUG Deleted.

Nothing in this list was dropped as a live path. Remaining references are only in hands-off trees.

Proof

rg of the deleted knobs/modules/functions. Hits left outside docs/ and bench_results/:

  • VLLM_BT_DEBUG: vllm/v1/worker/gpu/model_runner.py, vllm/v1/attention/backends/rdna_attn.py (hands off).
  • VLLM_QSA_RDNA2_DEBUG: csrc/rocm/qsa_rdna2.cu printf (hands off).
  • awq_gemm_rdna2_prefill: string inside the kernel-name regex in benchmarks/kernels/w4a16_compile_dispatch/probe.py (not a call; not edited).
  • Historical mentions of VLLM_LOG_GDN_PTRS, _awq_prefill_available, and rocm_graph_keepalive.py under docs/ only.

Zero hits in vllm/**/*.py for: VLLM_LOG_GDN_PTRS, VLLM_GDN_OUT_DEBUG, VLLM_GDN_NAN_DEBUG, VLLM_GDN_STATE_DEBUG, VLLM_GDN_SAVE, VLLM_GDN_DBG, VLLM_MAMBA_COPY_DEBUG, VLLM_QSA_RDNA2_DEBUG, _qsa_dump_bad_rows, VLLM_PIECE_IN_DEBUG, VLLM_CG_NAN_INPUT_CHECK, VLLM_CG_REPLAY_LOG, VLLM_PIECE_DUMP, VLLM_PIECE_TP, VLLM_W4A16_PTR_DEBUG, _awq_prefill_available, VLLM_DEBUG_PREFIX_HIT, VLLM_MOE_NAN_DEBUG, VLLM_LAYER_NAN_DEBUG, VLLM_PLE_NAN_DEBUG, rocm_poolwatch, rocm_graph_keepalive.

.venv/bin/python -m py_compile <12 touched .py files>
PY_COMPILE_OK

Ruff 0.14.0 (repo pre-commit rev) on the touched files. No new diagnostics versus rdna_extras HEAD. Remaining ruff check hits are pre-existing on untouched lines (HEAD had more: the deleted probes were the extra E501/UP031/SIM105/B018):

  • backends.py E501 line 714 (same line on HEAD)
  • cuda_graph.py SIM102 in kept _nan_to_num_tree
  • rdna2_w4a16.py B018 on the kept torch.ops._rocm_C.w4a8_gemm_rdna2 probe
  • qwen_gdn_linear_attn.py I001 plus two pre-existing E501s
  • qsa_rdna2.py I001 (pre-existing import gap)

ruff format --check: 7 touched files already formatted (communication_op.py, ple_offload_layer.py, model.py, kv_cache_coordinator.py, block_table.py, mamba_utils.py, utils.py). The other 5 also fail format --check on HEAD; they were not reformatted, so surviving lines stay as they were.

Diff: 14 files, 4 insertions, 985 deletions. The four insertions restore the pre-probe return / gptq fallback (elif became if after the AWQ arm was removed).

Test Plan

  • python -m py_compile on every touched file
  • ruff check and ruff format --check with ruff 0.14.0 and pyproject.toml
  • rg for each deleted knob
  • CPU pytest, if the modules import without a ROCm build:
    • tests/compile/test_cudagraph_replay_inputs.py
    • tests/kernels/quantization/test_rdna2_w4a16_selection.py
    • tests/v1/core/prefix_cache/test_partial_prefix_cache_hits.py

Test Result

py_compile: pass.

Ruff: see Proof. No new findings versus rdna_extras.

Pytest (CPU torch 2.13.0+cpu, PYTHONPATH=/workspace, no ROCm extension build):

6 failed, 67 passed, 11 skipped, 73 errors in 11.51s

Imports succeeded without a compiled ROCm extension. import_kernels only warns when vllm._C is missing.

  • 11 skipped are the GPU/ROCm gates (CUDA/HIP required, ROCm only), including the three VLLM_CG_SKIP_REPLAY capture/replay cases.
  • 73 errors are teardown only: cleanup_dist_env_and_memory calls torch.accelerator.empty_cache() because platform detection does not see this tree as CPU (vLLM package was not found when importing from PYTHONPATH). The test bodies of the W4A16 selector cases (test_rdna2_w4a16_inner_dispatch, registry order, backend map, compile dispatch) ran and were not assertion failures.
  • 6 failures are not from this diff:
    • test_should_copy_and_wrap_eager_piecewise_graphmodules and test_mrope_get_positions_contiguous_per_capture_size raise Failed to infer device type / pin_memory=True with no accelerator.
    • four test_mamba_align_split_* cases raise AttributeError: 'types.SimpleNamespace' object has no attribute '_effective_lpt' in vllm/v1/core/sched/scheduler.py, which this PR does not touch.

Model eval: not applicable. No kernel math, sampler, or default serving path changed.

Open in Web Open in Cursor 

Remove owner-only forensics that write /tmp logs, print, torch.save,
or host-scan for NaNs. Env-unset paths keep the same kernel selectors
and copy-mismatch fallback.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
@BlivionIaG BlivionIaG closed this Oct 7, 2026
@BlivionIaG

Copy link
Copy Markdown
Author

Closed as duplicate of #44. Same Python bring-up /tmp probe deletions. Prefer #44.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants