Skip to content

fix(nvarc): bound and reap interpreter processes - #3054

Open
ananthsub wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
ananthsub:ananthsub/fix-nvarc-process-lifecycle
Open

fix(nvarc): bound and reap interpreter processes#3054
ananthsub wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
ananthsub:ananthsub/fix-nvarc-process-lifecycle

Conversation

@ananthsub

Copy link
Copy Markdown
Contributor

Summary

  • cap inductive Python subprocesses per server worker with a positive python_max_concurrency setting
  • terminate, escalate to kill, and wait for children on timeout or caller cancellation before returning
  • expose queue and execution time as separate verify spans and document timeout admission semantics
  • replace duplicated test implementations with production-path tests covering success and every failure lifecycle

Correctness and reliability

The configured execution timeout starts after semaphore admission, so queueing does not consume a submission's runtime budget. Existing None prediction and zero-reward behavior is preserved for execution failures. This change makes no throughput improvement claim; admission control intentionally trades queue latency for bounded resource use.

Validation

  • uv run --extra dev pytest resources_servers/nvarc/tests/test_app.py -q — 43 passed
  • scoped pre-commit — passed
  • real-process stress: 32 infinite-loop submissions at concurrency 8 produced peak 8 children, zero survivors after quiescence, and 4.11 s elapsed
  • repeated-failure test checks both descendants and open file descriptors after 32 subprocess failures

Closes #3029

@ananthsub

Copy link
Copy Markdown
Contributor Author

fyi @esarafian

Comment thread resources_servers/nvarc/app.py Outdated
input_json=json.dumps(input_grid),
)

proc = await asyncio.create_subprocess_exec(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this stay inside the try?
If spawning raises OSError (for example, process or file-descriptor exhaustion), verification now returns a 500 instead of the previous None

Prevent timed-out or cancelled inductive verification children from surviving while limiting per-worker interpreter fanout.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@ananthsub
ananthsub force-pushed the ananthsub/fix-nvarc-process-lifecycle branch from b5141b0 to b281ae0 Compare September 3, 2026 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bound and clean up nvarc interpreter processes

2 participants