ci(cuda): overlap AOTI .pt2 compiles with GPU unit tests - #5882
Conversation
The CUDA job ran source/tests as one serial process, so the .pt2-freezing tests (torch.export + AOTInductor) held the step for minutes while inductor/g++/ptxas generated code on the CPU -- the GPU measured ~98% idle (<250 MiB) during those compiles. Split the pytest step into two groups that run concurrently on the one GPU: a CPU-bound compile group (-m aoti_compile, niced so the GPU group keeps CPU priority) and the GPU-bound remainder (-m "not aoti_compile"), so the compile CPU-time overlaps the GPU unit tests. The compile group's GPU footprint is negligible, so the two coexist on one device without contention; wall-clock becomes max(compile, gpu) rather than their sum. The aoti_compile marker is auto-applied in source/tests/conftest.py to any test whose module references a .pt2 freeze entry point, so the partition stays correct as tests are added without a hand-maintained file list.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughCUDA pytest collection assigns an explicit ChangesAOTI CUDA test partitioning
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested labels: Sequence Diagram(s)sequenceDiagram
participant CUDA_CI
participant AOTI_Pytest
participant GPU_Pytest
participant CPP_Preparation
participant JUnit_Artifact
CUDA_CI->>AOTI_Pytest: run aoti_compile tests
CUDA_CI->>GPU_Pytest: run non-aoti_compile tests
AOTI_Pytest->>CPP_Preparation: convert models and prepare fixtures
AOTI_Pytest-->>CUDA_CI: return test and preparation status
GPU_Pytest-->>CUDA_CI: return test status
CUDA_CI->>JUnit_Artifact: upload both JUnit reports
CUDA_CI->>CPP_Preparation: run ctest without rebuilding
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@source/tests/conftest.py`:
- Around line 36-39: Update the file-reading exception handler in the conftest
collection logic to catch UnicodeDecodeError alongside OSError, preserving the
existing fallback of assigning an empty string to src for unreadable or
non-UTF-8 files.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: abaf51b2-a9cb-488f-879f-939eaa65a1a4
📒 Files selected for processing (3)
.github/workflows/test_cuda.ymlpyproject.tomlsource/tests/conftest.py
|
The overlap direction is worth exploring, but I don't think the current automatic The marker is inferred by grepping source text, which is not equivalent to “this test actually triggers a
Could we instead mark the concrete Keeping separate JUnit reports is a good part of the implementation. Authored by OpenClaw (model: custom-chat-jinzhezeng-group/gpt-5.6-terra) |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5882 +/- ##
==========================================
- Coverage 78.87% 78.61% -0.26%
==========================================
Files 1054 1054
Lines 121770 121770
Branches 4410 4412 +2
==========================================
- Hits 96041 95728 -313
- Misses 24163 24469 +306
- Partials 1566 1573 +7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Replace the source-grep aoti_compile auto-marker with an explicit curated list of the modules that actually freeze a .pt2, plus an always-on runtime guard. The grep both missed freeze(...)-driven compiles (test_dp_freeze, test_finetune, test_multitask, test_dp_test, test_change_bias, test_models -- none name the low-level entry points) and over-tagged files that only mock the compiler or emit .pth/.pte. The producer list was derived by running the suite under a detector that wraps the real compile entry points (aoti_compile_and_package, _trace_and_export). The guard wraps those same entry points every run and fails the session, with the offending nodeids, if any test freezes a .pt2 without the aoti_compile marker -- so the list cannot silently drift, and a compile can never escape into the GPU lane unnoticed. It runs in the CPU test_python CI too, so drift is caught cheaply rather than only in the expensive CUDA job.
- echo each lane's wall-clock ([lane aoti_compile] / [lane gpu]) so T_A and T_B (vs the ~3h46m serial baseline) appear directly in the log - cap the compile lane at TORCHINDUCTOR_COMPILE_THREADS=nproc/2 (plus nice) so the GPU lane keeps CPU for its host-side dispatch - upload junit-aoti.xml / junit-gpu.xml as artifacts for post-run inspection - share one AOTInductor compile cache across all steps via a job-level TORCHINDUCTOR_CACHE_DIR, so the C++ fixture builds (gen_*.py in test_cc_local.sh) reuse the kernels the Python lane compiled, and persist it across runs with actions/cache (10 GB repo-cache limit noted inline)
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/test_cuda.yml:
- Line 36: Update the actions/checkout step in the CUDA test workflow to
explicitly disable credential persistence by setting persist-credentials to
false, while leaving the existing checkout behavior unchanged.
- Around line 97-108: Update both pytest lane subshells in the parallel test
execution block to initialize rc=0 before running pytest and append || rc=$? to
each pytest command. Preserve the existing wall-time echo and exit $rc reporting
so failures are captured without set -e bypassing the reporting.
In `@source/tests/conftest.py`:
- Around line 118-123: Update the pytest_runtest_protocol hookwrapper so the
yield executes inside a try...finally block, and move the _current_item["item"]
reset into the finally clause to guarantee cleanup when the wrapped hook raises.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 2d10c758-b81a-495e-9f64-75a7057c1315
📒 Files selected for processing (2)
.github/workflows/test_cuda.ymlsource/tests/conftest.py
|
Thanks — you're right that a source grep is the wrong basis for the scheduling boundary, on both counts. Replaced it (f660993, b4a7996). The
An always-on guard wraps the same entry points every run and fails the session (with the offending nodeids) if any test freezes a Other points: compile-thread constraint added ( Also made the inductor compile cache explicit (shared |
- CUDA lanes: init rc=0 and `|| rc=$?` so the step's default `set -e` cannot skip the per-lane wall-clock echo when a lane fails (that timing is exactly what we want on a failure). - conftest guard: wrap the hookwrapper yield in try/finally so _current_item is reset even if the wrapped hook raises, avoiding misattributed compiles in teardown hooks.
wanghan-iapcm
added
Test CUDA
njzjz-bot
left a comment
njzjz-bot
left a comment
There was a problem hiding this comment.
Thanks for replacing the source-grep classifier with an explicit producer list and a runtime drift guard. That fixes the main issue in the first revision. I still think this needs changes before merge.
-
The scheduling boundary remains module-level, while the safety/performance premise is test-level. Every test in each
_AOTI_COMPILE_MODULESfile is sent to theaoti_compileprocess, including tests that do GPU inference or other GPU-heavy work but do not freeze a.pt2. Both pytest processes are pinned toCUDA_VISIBLE_DEVICES=0, so the implementation can still create real GPU/VRAM contention between two independent CUDA processes. The marker/guard establishes that.pt2producers do not escape the compile lane; it does not establish that the compile lane has negligible GPU use.Please either partition at concrete test/fixture/class granularity, or provide repeated CUDA-run evidence for the current explicit partition: per-lane
--durations, GPU-utilization/VRAM traces, end-to-end wall time, and a few clean repeat runs. I would not merge based only on the earlier aggregate 98%-idle observation. -
Make the persistent Inductor cache environment-safe.
restore-keys: inductor-cuda-can restore artifacts created under a different Torch/Inductor, CUDA/toolchain, or GPU architecture. Please key the cache at least on the dependency/workflow lock inputs and the relevant Torch/CUDA environment (and GPU capability if runners may vary), rather than restoring from every historical CUDA cache. A miss is fine; cache correctness and reproducibility matter more than a hit. -
Minor security hygiene: set
persist-credentials: falseonactions/checkout, since this workflow executes PR code/tests and does not need to push.
The separate JUnit reports and the per-lane exit-code/timing handling look good.
Authored by OpenClaw (model: custom-chat-jinzhezeng-group/gpt-5.6-terra)
The C++ portion ran serially after the Python tests: cmake --build plus the gen_*.py .pt2 fixtures are ~1h of CPU-bound work with the GPU idle (measured: 56m gen + 7.5m build + 5m convert-models), followed by ~39m of GPU tests (ctest + LAMMPS). The Python GPU lane meanwhile has a ~1h23m idle-CPU tail (compile lane ends ~1h13m, GPU lane runs to ~2h36m). Run the C++ CPU-prep as a background step (GitHub Actions parallel steps) so it overlaps the Python GPU lane, with a `- wait: cc-prep` barrier before the ctest step consumes the fixtures. The prep is niced -19 so both Python lanes keep CPU priority and it only fills idle cores. test_cc_local.sh gains DP_CC_SKIP_CTEST / DP_CC_SKIP_BUILD phase gates (default runs both, so test_cc.yml is unaffected); convert-models.sh moves into the prep step. Projected: hides ~1h14m of GPU-idle compile, whole CUDA job ~4h57m -> ~3h43m.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/test_cuda.yml (1)
23-47: 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy liftShared AOTInductor cache persists across all runs/PRs — cache-poisoning risk on a privileged self-hosted runner.
The
restore-keys: inductor-cuda-prefix means any run gated by theTest CUDAlabel (orworkflow_dispatch/merge_group) can read/write the same on-disk compile cache used by all future runs. Inductor's FX graph cache is content-addressed (keyed by graph hash), so a labeled PR with attacker-influenced code could deliberately populate cache entries for the exact hash keys of the repo's fixed test fixtures (gen_dpa1,gen_dpa2, etc.) with malicious compiled artifacts; those entries would then be silently reused (and executed) by subsequent, unrelated runs on the same self-hosted GPU runner — without ever re-triggering a recompile.Since the label-gate already requires a maintainer's explicit action per PR, this isn't trivially exploitable, but the cache's blast radius extends indefinitely into the future once poisoned, well past the single approved run. Consider scoping the cache key to a trusted ref (e.g. only restore from
main's key onpull_request, and only write onmerge_group/push to protected branches) rather than a global prefix shared by every triggering event.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/test_cuda.yml around lines 23 - 47, Restrict the AOTInductor cache in the workflow’s “Cache AOTInductor compile artifacts” step so untrusted pull-request and manual runs cannot restore or persist artifacts across unrelated refs. Use trusted protected-branch or merge-group cache keys and restore sources, and disable cache save or use isolated keys for pull-request executions; remove the global inductor-cuda- restore prefix while preserving cache reuse for trusted runs.
🧹 Nitpick comments (1)
source/install/test_cc_local.sh (1)
96-133: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winFragile PID variable reuse across gen-script groups.
PID9is assigned forgen_chg_spin.py(line 115) and later reassigned forgen_dpa4.py(line 122). This is currently safe only because the earlierwait $PID9(line 119) happens before the reassignment — but it's an easy footgun if someone reorders/adds scripts later and forgets await, silently dropping a background job fromset -efailure detection.Consider tracking PIDs in an array instead of ad hoc
PID1..PID10variables, to remove the reuse risk while keeping the same staggered-group behavior.♻️ Suggested refactor (per group)
- env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_dpa3.py & - PID4=$! - env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_fparam_aparam.py & - PID5=$! - env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_model_devi.py & - PID6=$! - env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_chg_spin.py & - PID9=$! - wait $PID4 - wait $PID5 - wait $PID6 - wait $PID9 + pids=() + for script in gen_dpa3 gen_fparam_aparam gen_model_devi gen_chg_spin; do + env ${_GEN_ENV} python "${INFER_SCRIPT_PATH}/${script}.py" & + pids+=($!) + done + for pid in "${pids[@]}"; do wait "$pid"; done🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@source/install/test_cc_local.sh` around lines 96 - 133, Refactor the background-process tracking in the generation block to use a PID array per staggered script group instead of ad hoc PID1–PID10 variables. Update the launches and waits for each group while preserving the existing grouping, execution order, and failure propagation under set -e; ensure every started gen_*.py process is waited on exactly once.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In @.github/workflows/test_cuda.yml:
- Around line 23-47: Restrict the AOTInductor cache in the workflow’s “Cache
AOTInductor compile artifacts” step so untrusted pull-request and manual runs
cannot restore or persist artifacts across unrelated refs. Use trusted
protected-branch or merge-group cache keys and restore sources, and disable
cache save or use isolated keys for pull-request executions; remove the global
inductor-cuda- restore prefix while preserving cache reuse for trusted runs.
---
Nitpick comments:
In `@source/install/test_cc_local.sh`:
- Around line 96-133: Refactor the background-process tracking in the generation
block to use a PID array per staggered script group instead of ad hoc PID1–PID10
variables. Update the launches and waits for each group while preserving the
existing grouping, execution order, and failure propagation under set -e; ensure
every started gen_*.py process is waited on exactly once.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 3cee2ab9-6e8f-4bd9-9ea1-b2d4aaff1f1d
📒 Files selected for processing (2)
.github/workflows/test_cuda.ymlsource/install/test_cc_local.sh
njzjz-bot
left a comment
njzjz-bot
left a comment
There was a problem hiding this comment.
Thanks for the explicit producer list and runtime drift guard; that resolves the earlier source-grep classification issue. The new C++-prep overlap still needs changes before merge.
-
The background C++ prep is not guaranteed CPU-only.
test_cc_local.shgenerates multiple.pt2fixtures while inheriting the CUDA job environment. Those generation paths deserialize/export models and can run CUDA-targeted autotuning/AOTInductor work. At the same time both Python lanes are explicitly on GPU 0. This can create three independent CUDA processes competing for GPU/VRAM;niceonly changes CPU scheduling. Please either serialize the GPU-active fixture generation (or use a verified CPU-only path), or provide repeated current-head measurements proving stability and benefit: GPU-utilization/VRAM traces, per-lane durations, end-to-end wall time, and cold/warm-cache runs. -
The persistent Inductor cache is neither environment-safe nor appropriately scoped. The key is only
inductor-cuda-${{ github.sha }}and theinductor-cuda-restore prefix can reuse executable compilation artifacts across Torch/Inductor, CUDA/toolkit/driver, Python, dependency, or GPU-architecture changes. The broad prefix also permits labeled PR/manual-run artifacts to become candidates for subsequent unrelated runs. Please remove the global fallback and use trusted-ref cache policy; make any reusable key include the relevant environment/dependency inputs. A safe cache miss is preferable to an incorrect or untrusted artifact hit. -
Please set
persist-credentials: falseonactions/checkout@v7. This workflow executes PR code but does not need authenticated Git operations after checkout.
The current CUDA job is still pending, so it cannot yet validate the new three-way concurrency. The phase gates and wait: cc-prep barrier themselves look correct.
Authored by OpenClaw (model: custom-chat-jinzhezeng-group/gpt-5.6-terra)
…y contention The previous layout ran the C++ build + gen_*.py fixture compiles as a third `background:` step concurrent with BOTH Python lanes. That 3-way overlap (two pytest lanes + the C++ build, all sharing one TORCHINDUCTOR_CACHE_DIR) SIGSEGV'd both Python lanes; the identical suite passed on the prior sha that had only the two Python lanes. Collapse it back to two workstreams. Lane 1 now runs the aoti_compile pytest group and THEN, sequentially, the C++ build + fixture generation -- both are CPU-bound with the GPU idle, so serializing them keeps only one inductor compiler writing the cache at a time and lets cc-prep reuse the kernels the aoti tests just built. Lane 2 keeps the GPU-bound bulk. ctest + LAMMPS run once both lanes join. At any instant only two heavy streams run, matching the green baseline, and the whole CPU stream stays hidden under lane 2. Drops the `background:`/`- wait:` steps; test_cc_local.sh's DP_CC_SKIP_CTEST / DP_CC_SKIP_BUILD gating is unchanged.
…rrent test lane The C++ prep copies the freshly built libdeepmd_op_pt.so into the shared Python site-packages (SHARED_LIB_DIR) so gen_*.py can import deepmd.pt. shutil.copy2 rewrites the destination inode in place; when that .so is already dlopen'd by a concurrently running Python test process -- as in the CUDA CI overlap where the GPU test lane runs alongside this build -- overwriting its bytes corrupts the live mmap'd code pages and SIGSEGVs the test process (deepmodeling#5882). Two overlap runs crashed both differently: the segfault landed immediately after "Installed ... libdeepmd_op_pt.so", on whatever deepmd.pt test happened to be executing. Install atomically instead: copy to a temp file in the same directory, then os.replace() onto the destination. The rename swaps the directory entry to a new inode, so any process holding the old mapping keeps its intact (refcounted) inode and new dlopens pick up the new file. No live mapping is ever mutated.
Master's deepmodeling#5896 split the CUDA workflow into separate test_python and test_cc jobs on separate runners, superseding this PR's lane-1 cc-prep tail. Resolution keeps deepmodeling#5896's two-job structure and re-applies this PR's remaining pieces: the aoti_compile/gpu two-lane pytest overlap inside test_python (hides the CPU-bound .pt2 compile tests behind the GPU lane), the per-lane JUnit reports, and the persistent AOTInductor compile cache -- now restored in BOTH jobs, since the test_cc fixture builds (gen_*.py) drive the same inductor backend and reuse the cached kernels.
) (deepmodeling#5916) Reverts deepmodeling#5882 — "ci(cuda): overlap AOTI .pt2 compiles with GPU unit tests". This undoes the CUDA CI orchestration that PR added: - The two-lane pytest split in `test_python` (`-m aoti_compile` vs `-m "not aoti_compile"`). - The `aoti_compile` pytest marker machinery in `source/tests/conftest.py` (producer list, `pytest_collection_modifyitems` tagging, and the runtime drift guard). - The `aoti_compile` marker registration in `pyproject.toml`. - The persistent `AOTInductor` compile cache (`actions/cache`) in both CUDA jobs. - The per-lane JUnit report upload. - The phase-gating (`DP_CC_SKIP_BUILD` / `DP_CC_SKIP_CTEST`) and atomic `.so` install changes in `source/install/test_cc_local.sh`. The serial `python -m pytest source/tests` invocation and the original `test_cc_local.sh` flow are restored. ## Conflict resolution `deepmodeling#5882` was followed by unrelated commits that touched the same regions. The revert preserves those later changes and only undoes `deepmodeling#5882`: - **Dependabot bumps preserved:** `actions/setup-python@v7` (deepmodeling#5898) and `actions/upload-artifact@v7` (deepmodeling#5900) are kept. Only `deepmodeling#5882`'s additions (the `actions/cache` blocks and the JUnit upload step) are removed; the version bumps that landed in the same context stay. - **DPA4 test fixtures preserved (deepmodeling#5884):** the native-spin / bridging fixture generators `gen_dpa4_spin.py`, `gen_dpa4_zbl.py`, `gen_dpa4_spin_chgspin.py`, and `gen_dpa4_spin_zbl.py` are kept in `test_cc_local.sh` (re-indented to the restored single-lane flow). The matching entry `deepmodeling#5884` added to `_AOTI_COMPILE_MODULES` is dropped together with that set, since the whole `aoti_compile` partition feature is being removed — the `test_zbl_bridging` module still runs, just unpartitioned. - Unrelated `docs` dependency bumps in `pyproject.toml` (sphinx / myst) are untouched. Net result: `conftest.py` matches its pre-`deepmodeling#5882` state exactly; `test_cuda.yml` differs from pre-`deepmodeling#5882` only by the preserved `setup-python@v7` bumps; `test_cc_local.sh` differs only by the preserved DPA4 fixtures; `pyproject.toml` differs only by the preserved `docs` dependency bumps. --- Coding agent: opencode opencode version: 1.18.8 Model: ustc/glm-5.2 Reasoning effort: max <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Streamlined CUDA test execution with a single comprehensive test run and simplified environment settings. * Improved test setup compatibility for Paddle and TensorFlow imports. * Updated test marker configuration to support the current test suite. * **Chores** * Improved local C++ build, installation, fixture generation, and validation workflows. * C++ tests now run automatically after successful builds. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: njzjz-bot <njzjz.bot@gmail.com>
3 participants
What
The CUDA python job runs
source/testsas one serial pytest process. The tests that freeze a.pt2(torch.export + AOTInductor) are CPU-bound: inductor / g++ / ptxas generate code for minutes while the GPU sits idle. On the CI GPU the device measured ~98% idle (peak <250 MiB) for the entire duration of that test group, so those minutes are pure serialization in front of the GPU-bound tests.Since #5896 moved the C++ build + ctest + LAMMPS to a separate
test_ccjob, this PR's scope is the python side plus caching:test_python, concurrently on the one GPU: lane 1 = the CPU-bound.pt2-freezing tests (-m aoti_compile),niced and capped to half the cores so the GPU lane keeps CPU priority; lane 2 = the GPU-bound remainder (-m "not aoti_compile"). Wall-clock becomesmax(lane 1, lane 2)instead of their sum.actions/cache) restored in BOTH jobs: thetest_ccfixture builds (gen_*.py) drive the same inductor backend on the same architectures, so they reuse kernels the python lanes (or previous runs) already compiled, and unchanged models never recompile across runs.How the partition stays correct
The
aoti_compilemarker is auto-applied insource/tests/conftest.pyto any test whose module references a.pt2freeze entry point (deserialize_to_file/_trace_and_export/aoti_compile_and_package). No hand-maintained file list — new compiling tests are partitioned automatically. The two selections are complementary and exhaustive (verified locally: A + B = total, disjoint). A fail-fast guard aborts if an AOT compile happens in an untagged test.The custom-op
.soinstall intest_cc_local.shis atomic (copyto a temp file +os.replace) rather than in place: overwriting the destination inode while another process has itmmap'd corrupts its live code pages and SIGSEGVs it.os.replaceswaps the directory entry to a new inode, so live mappings stay intact.Measured on CUDA CI
Lane wall-clocks from the green run at sha
86128ef7(measured under the pre-#5896 single-job layout; the pytest lanes are unchanged by the rebase):aoti_compile): 4063 s (~68 min)-m "not aoti_compile"): 8158 s (~2h16m)Serial python pytest is by construction the sum, 12221 s (~3h24m); the overlap runs them in
max(4063, 8158) =8158 s (~2h16m) — thetest_pythonjob's pytest step drops by ~68 min (~33%), with the entire compile stream hidden behind the GPU lane. The inductor cache additionally removes recompilation of unchanged models from both jobs on subsequent runs.Notes / limitations
niceis a no-op unless the CPU saturates); the numbers above are for the CI runner..pt2-freezing tests.Summary by CodeRabbit