fix #161: XQA cubin generation ignores resolved output dir under spawn/forkserver - #177
Open
Hi5808 wants to merge 1 commit into
Open
fix #161: XQA cubin generation ignores resolved output dir under spawn/forkserver#177Hi5808 wants to merge 1 commit into
Hi5808 wants to merge 1 commit into
Conversation
…r spawn/forkserver gen_cubins.py's worker pool (multiprocessing.Pool.map) reads the module-scope `cubin_dir`/`nvcc_bin`/`clean_cubin` globals. These are only rebound to their CLI-resolved values inside `if __name__ == "__main__":`, which worker processes never execute when the multiprocessing start method re-imports the module (spawn/forkserver) instead of forking. Under fork (Linux's current default) this is masked because children inherit the parent's post-__main__ state, but it silently falls back to the relative module-level default elsewhere, matching the ptxas "could not be opened" failure reported in NVIDIA#161. Pass the resolved values into each worker explicitly via Pool(initializer=...) so behavior no longer depends on the start method. Signed-off-by: Kino <hanauma5@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the latent bug identified in #161 (separate from the permissions/OS-specific
failure discussed there).
gen_cubins.py's worker pool reads the module-scopecubin_dir/nvcc_bin/clean_cubinglobals, which are only rebound to theirCLI-resolved values inside
if __name__ == "__main__":. Worker processes neverexecute that block when the multiprocessing start method re-imports the module
(
spawn/forkserver) instead of forking, so they silently fall back to therelative module-level default for
cubin_dirinstead of the resolved--output_dir.Under
fork(Linux's current default) this is masked because child processesinherit the parent's post-
__main__state — which is why it hasn't causedvisible failures so far. It will start affecting
fork-based setups too oncePython 3.14 switches Linux's default start method to
forkserver.Changes
init_cubin_gen_worker()and pass it tomultiprocessing.Poolviainitializer=/initargs=, so each worker gets the resolvedcubin_dir/nvcc_bin/clean_cubinexplicitly rather than relying oninherited module state.
kernelSrcs/is excluded from this repo's pre-commit hooks, so no formattingchanges were needed beyond the diff itself.
Test plan
python3 -m py_compile kernelSrcs/xqa/gen_cubins.pyforkmultiprocessingstart method (e.g.
multiprocessing.set_start_method("spawn")) to confirmthe fix, since I don't have a repro environment for that start method on
this hardware (Python 3.12.3 on Jetson Orin NX defaults to
fork).