test(pt): reuse PT2 artifacts for metadata checks - #5812
Conversation
Build the base AOTInductor package once in a module-scoped fixture, derive metadata variants by editing ZIP extras, and share the artifact between metadata and inference test classes.\n\nCoding-Agent: Codex\nCodex-Version: codex-cli 0.144.4\nModel: gpt-5.6-sol\nReasoning-Effort: xhigh
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe deep evaluation tests replace per-class artifact setup with shared pytest fixtures. Helpers create PT2/PTE variants, metadata assertions use shared archives, and deep evaluation tests consume fixture-provided artifact paths. ChangesDeep evaluation artifact tests
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5812 +/- ##
==========================================
- Coverage 79.21% 78.96% -0.25%
==========================================
Files 1069 1069
Lines 124070 124070
Branches 4522 4527 +5
==========================================
- Hits 98278 97969 -309
- Misses 24171 24484 +313
+ Partials 1621 1617 -4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
The CI timing shows that this optimization is effective for the targeted tests, but the workflow-level benefit is limited. Comparing the Python 3.10 duration artifacts from the base SHA
Conclusion: the PR is a valid targeted optimization, with a measured ~16% reduction for the affected tests in this CI run, but it should not be interpreted as compiling the fixture only once across CI or as an established whole-workflow speedup. It would be useful to recheck after the updated duration cache has rebalanced the splits. Coding agent: Codex |
|
Possible reviewers based on changed lines, exact file history, and exact-file review history:
No review request was made automatically. Coding agent: Codex |
|
Moving this PR back to draft because the current CI result does not demonstrate an overall CI speedup. The targeted 22 PT2 tests improved from 2306.7 s to 1942.0 s of cumulative test time (15.8%), but the CI command uses 12 pytest splits. The module-scoped fixture is local to each pytest process, so the PT2 artifact was still compiled independently in 10 shards rather than once. In the available paired run, the cumulative duration of the 12 main pytest steps increased from 36,727 s to 38,092 s, and the slowest shard increased from 3,426 s to 3,594 s. Runner noise means this does not prove a regression, but it also does not establish the intended CI acceleration. Before marking this ready, the tests should either share a prebuilt artifact across shards, be kept in a dedicated/single shard, or otherwise be validated with repeated CI runs showing a shorter critical path. Coding agent: Codex |
Summary
Validation
Coding agent: Codex
Codex version: codex-cli 0.144.4
Model: gpt-5.6-sol
Reasoning effort: xhigh
Summary by CodeRabbit