[CI] externalize test config and add /nightly dispatch resolver#10614
[CI] externalize test config and add /nightly dispatch resolver#10614zhangxinyuehfad wants to merge 2 commits into
Conversation
zhangxinyuehfad
requested review from
LCAIZJ,
Yikun and
wangxiyuan
as code owners
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request refactors the CI nightly testing infrastructure by centralizing test configurations into a single YAML file and introducing a dynamic dispatch mechanism. By decoupling the test matrix from hardcoded workflow files, the system now allows for more flexible and PR-specific test execution via slash commands, simplifying the process for developers to validate new test cases against real hardware. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request externalizes the nightly test configurations into a centralized YAML file (nightly_config.yaml) and removes the old accuracy_groups_a2.json file. It also updates the developer guide to document the new test matrix data source, instructions for adding new test cases, and updated troubleshooting steps. The reviewer noted that the PR title and description do not adhere to the repository's style guide and provided a suggested title and summary to fix this.
| | Code tested | Pre-built nightly image | Your PR's HEAD commit (source installed fresh) | | ||
| | Test scope | All tests | Configurable via `/nightly <names>` | | ||
| | vLLM + vllm-ascend | From image | Checked out and installed from source | | ||
| | Test matrix | From main branch's matrix YAML | From PR branch's matrix YAML | |
There was a problem hiding this comment.
The Pull Request title and description do not fully adhere to the repository's Pull Request Summary Style Guide. Specifically, the PR title is missing the required Action prefix, and the description sections for user-facing changes and testing are empty.
Please update them as suggested below:
Suggested PR Title:
[CI][Feature] externalize test config and add /nightly dispatch resolverSuggested PR Summary:
### What this PR does / why we need it?
- Move per-platform test configs (A2/A3 single-node, multi-node, double-node, multi-card, accuracy) into a centralized `.github/workflows/configs/nightly_config.yaml` and replace the old `accuracy_groups_a2.json` with the new schema.
- Update `resolve_nightly_tests.py` to support both the **matrix mode** (read sections from `nightly_config.yaml`) and a new **dispatch mode** (resolve `/nightly <name>` tokens to a2/a3 dispatch flags + transformed test cases).
- Wire `pr_nightly_command.yml` to consume the dispatch outputs so `/nightly` slash commands can route PR-based runs to the right nightly workflow with proper PR-ref sparse-checkout.
- Update the developer guide to document the new flow.
- Consume matrix references from generate-a2/a3-matrix jobs instead of inline `test_config` matrices.
- Add generate-a2/a3-matrix to the corresponding `needs` arrays.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Tested via CI runs and verifying the `/nightly` command dispatch logic.References
- The PR title and summary must follow the specified formats in the Pull Request Summary Style Guide. (link)
github-actions
Bot
added
documentation
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
zhangxinyuehfad
force-pushed
the
zxy_fix_nightly
branch
from
f912837 to
bb64abe
Compare
zhangxinyuehfad
force-pushed
the
zxy_fix_nightly
branch
5 times, most recently
from
4a6a278 to
1ae828c
Compare
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
zhangxinyuehfad
force-pushed
the
zxy_fix_nightly
branch
from
1ae828c to
7e51e8e
Compare
Replace inline test_config matrices in single-node-tests, multi-node-tests, double-node-tests, multi-card-tests, and single-node-accuracy-tests with references to outputs from the generate-a2/a3-matrix jobs. Add generate-a2/a3-matrix to the corresponding needs arrays. Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Move per-platform test configs (A2/A3 single-node, multi-node, double-node, multi-card, accuracy) into a centralized .github/workflows/configs/nightly_config.yaml and replace the old accuracy_groups_a2.json with the new schema. Update resolve_nightly_tests.py to support both the matrix mode (read sections from nightly_config.yaml) and a new dispatch mode (resolve /nightly <name> tokens to a2/a3 dispatch flags + transformed test cases). Wire pr_nightly_command.yml to consume the dispatch outputs so /nightly slash commands can route PR-based runs to the right nightly workflow with proper PR-ref sparse-checkout. Update the developer guide to document the new flow. Signed-off-by: hfadzxy <starmoon_zhang@163.com>
zhangxinyuehfad
force-pushed
the
zxy_fix_nightly
branch
from
7e51e8e to
1d60398
Compare
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
1 participant
What this PR does / why we need it?
.github/workflows/configs/nightly_config.yamland replace the oldaccuracy_groups_a2.jsonwith the new schema.resolve_nightly_tests.pyto support both the matrix mode (read sections fromnightly_config.yaml) and a new dispatch mode (resolve/nightly <name>tokens to a2/a3 dispatch flags + transformed test cases).pr_nightly_command.ymlto consume the dispatch outputs so/nightlyslash commands can route PR-based runs to the right nightly workflow with proper PR-ref sparse-checkout.test_configmatrices.needsarrays.Does this PR introduce any user-facing change?
How was this patch tested?