[CICD] Simplified workflows, Add lint_check & functional_tests, Switch to FlagCICD metax runner by BrianPei · Pull Request #29 · flagos-ai/Megatron-LM-FL

BrianPei · 2026-04-17T03:46:13Z

Description

Add CI/CD workflow support for multi-platform (CUDA & Metax) linting, functional testing, and log reporting. With significant workflow simplification, and Metax platform use BAAI runner configs.

Type of change

New feature (non-breaking change which adds functionality)
Infra/Build change (changes to CI/CD workflows or build scripts)
Code refactoring
Bug fix
Documentation change
Breaking change

Changes

CI workflow simplification: Consolidated and simplified functional_tests_common.yml, unit_tests_common.yml, and all_tests_common.yml
BAAI runner support: Added Metax platform config with BAAI runner
Linting: Added lint_common.yml with pylint integration (score ≥ 9.0)
Functional tests: Added GPT inference and GRPO functional test cases that run without external data/checkpoints; extended CUDA GPT mock-data coverage (recompute, SwiGLU, disable-bias, TP overlap)
Metax test compatibility: Curated ignored_tests list for >2 GPU / CUDA-only cases; limited functional coverage to validated 2-GPU GPT cases
Code fixes: Made RL/inference dependencies lazy/optional; fixed static inference token scheduling and position embedding for tiny models; handled missing optimizer group compatibility flags; declared six as runtime dependency
Cleanup: Removed 20+ unused upstream NVIDIA workflows; added PR template

Checklist

I have read and followed the contributing guidelines
The functionality is complete
I have commented my code, particularly in CI workflow setup steps
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added/updated tests that prove my feature works on CUDA and Metax platform
New and existing unit tests pass locally on CUDA and Metax platform

- Simplify functional_tests_common.yml: merge setup steps, add yq/envsubst/pybind11/uv-shim - Simplify unit_tests_common.yml: streamline environment setup and coverage - Fix all_tests_common.yml: always() condition for complete jobs - Add lint_common.yml with pylint integration - Add all_tests_cuda.yml with BAAI runner support - Update all_tests_metax.yml: enable unit+functional tests on BAAI runner - Update metax.yml: comprehensive ignored_tests for >2GPU/CUDA-only cases - Remove 20+ unused upstream NVIDIA workflows - Add PR template, remove obsolete CODEOWNERS/copy-pr-bot

- Add self-contained GPT inference and GRPO functional tests (no external data) - Add golden values for dev/dgx_a100 - Add CUDA GPT mock-data pretraining cases: regular, resume, recompute, SwiGLU, disable-bias, TP-overlap - Add multimodal LLaVA resume test case - Add yq_common.sh/yq_compat.py for self-contained yq resolution - Add mock null_numeric_agent for RL environments - Tune GRPO golden iteration-time thresholds for CI stability

- Fix static inference token scheduling and position embedding for tiny models - Fix GRPO self-contained inference batching fallback - Handle missing optimizer group compatibility flags - Make wandb, fastapi, RL server deps lazy/optional for test isolation - Declare six as base runtime dependency in pyproject.toml - Add unit tests for RL inference and text generation controller

Copilot

Pull request overview

This PR updates the project’s CI/CD and test infrastructure to run linting + unit/functional tests across CUDA and Metax (BAAI runners), while also adding several code fixes to make inference/RL flows and functional tests more self-contained and robust.

Changes:

Adds a shared lint_common.yml workflow and refactors CI entrypoints (all_tests_*, unit_tests_common, functional_tests_common) to simplify execution across platforms/runners.
Introduces new self-contained functional test cases (tiny GPT static inference + GRPO RL loop) plus expanded GPT/mock-data training matrix coverage and multimodal LLaVA mock-data coverage.
Fixes/adjusts inference + RL integration (static engine support in MegatronLocal, tokenizer compatibility, static context reset, attention mask behavior, optimizer param-group compatibility).

Reviewed changes

Copilot reviewed 77 out of 77 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/unit_tests/test_rl_inference_megatron.py	Adds async unit tests for `MegatronLocal` launch/generate paths.
tests/unit_tests/inference/text_generation_controllers/test_simple_text_generation_controller.py	Adds regression test for termination-token accounting.
tests/functional_tests/test_cases/multimodal-llava/multimodal_llava_mcore_te_tp1_pp1_resume_torch/model_config.yaml	Adds LLaVA ckpt-resume functional config (torch ckpt).
tests/functional_tests/test_cases/multimodal-llava/multimodal_llava_mcore_te_tp1_pp1_resume_torch/golden_values_dev_dgx_a100.json	Adds golden metrics for LLaVA resume case.
tests/functional_tests/test_cases/multimodal-llava/multimodal_llava_mcore_te_tp1_pp1/model_config.yaml	Updates LLaVA env vars (nondeterministic toggle).
tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_tiny_null_selfcontained/model_config.yaml	Adds self-contained tiny GPT static inference config.
tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_tiny_null_selfcontained/golden_values_dev_dgx_a100.json	Adds golden outputs for tiny GPT inference.
tests/functional_tests/test_cases/gpt/gpt_grpo_tp1_pp1_tiny_null_selfcontained/model_config.yaml	Adds self-contained tiny GRPO RL config.
tests/functional_tests/test_cases/gpt/gpt_grpo_tp1_pp1_tiny_null_selfcontained/golden_values_dev_dgx_a100.json	Adds golden metrics for GRPO loop.
tests/functional_tests/test_cases/gpt/gpt_grpo_tp1_pp1_tiny_null_selfcontained/env_config.yaml	Adds mock RL env config for GRPO functional run.
tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_mock_data_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml	Adds overlap+resume functional config.
tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_mock_data_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml	Adds overlap functional config.
tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_mock_data_swiglu/model_config.yaml	Adds pipeline+SwiGLU functional config.
tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_mock_data_resume_torch_dist_swiglu/model_config.yaml	Adds pipeline+SwiGLU resume config.
tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_mock_data_resume_torch_dist_disable_bias_linear/model_config.yaml	Adds disable-bias resume config.
tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_mock_data_disable_bias_linear/model_config.yaml	Adds disable-bias functional config.
tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_mock_data_uniform_full_recompute/model_config.yaml	Adds recompute functional config.
tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_mock_data_resume_torch_dist_uniform_full_recompute/model_config.yaml	Adds recompute resume config.
tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_mock_data_resume_torch_dist_dist_optimizer/model_config.yaml	Adds dist-optim resume config.
tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_mock_data_dist_optimizer/model_config.yaml	Adds dist-optim functional config.
tests/functional_tests/test_cases/gpt/_selfcontained/prepare_tiny_null_gpt_checkpoint.sh	Adds script to generate a tiny checkpoint for self-contained tests.
tests/functional_tests/shell_test_utils/yq_compat.py	Adds minimal yq-compatible YAML query/update helper.
tests/functional_tests/shell_test_utils/yq_common.sh	Adds yq resolution wrapper used by shell test utils.
tests/functional_tests/shell_test_utils/run_ci_test.sh	Switches to resolved yq, adds RL mode, tweaks args handling.
tests/functional_tests/shell_test_utils/_run_training.sh	Switches to resolved yq and adds extra distributed log listing.
tests/functional_tests/python_test_utils/test_grpo_training_loop.py	Loosens strict iteration-time gating for GRPO tiny loop.
pyproject.toml	Adds `six` to runtime dependencies.
pretrain_vlm.py	Extends `model_provider` signature and supports passing config/PG collection.
megatron/training/training.py	Makes param-group reordering tolerant of missing compatibility flags.
megatron/training/tokenizer/tokenizer.py	Adds pad/bos/eos/unk aliases for tokenizer compatibility.
megatron/rl/server/init.py	Removes heavy package-level imports to avoid optional deps at import time.
megatron/rl/rl_utils.py	Makes wandb typing optional; lazily imports HTTP server; fixes attention-mask slicing.
megatron/rl/inference/megatron.py	Adds static-engine path + locking and request building for local inference.
megatron/rl/inference/inference_interface.py	Adjusts relative import path.
megatron/rl/inference/api.py	Adjusts relative import path.
megatron/rl/agent/pass_at_evaluation_agent.py	Adjusts relative import path.
megatron/rl/agent/api.py	Adjusts relative import path.
megatron/plugin/utils.py	Makes `no_shared_fs` access safe via `getattr`.
megatron/core/transformer/transformer_layer.py	Disables BDA fusion during traced/static inference contexts.
megatron/core/transformer/mlp.py	Disables bias-activation fusion when inference mode is enabled.
megatron/core/transformer/dot_product_attention.py	Ensures fused softmax tracks runtime `attn_mask_type`.
megatron/core/tokenizers/text/libraries/null_tokenizer.py	Adds missing tokenizer API methods/properties for compatibility.
megatron/core/optimizer/optimizer.py	Makes param-group matching tolerant of missing flags/pre_* keys.
megatron/core/inference/text_generation_controllers/text_generation_controller.py	Fixes generated-length counting to include terminating token once.
megatron/core/inference/contexts/static_context.py	Clears KV cache on static context reset to avoid reuse across batches.
examples/rl/environments/mock/null_numeric_agent.py	Adds a mock RL agent for self-contained GRPO functional tests.
examples/rl/environments/mock/init.py	Adds package init for mock RL env module.
examples/inference/gpt/utils.py	Makes `simpy` optional; falls back to immediate enqueue offsets.
.github/workflows/unit_tests_common.yml	Simplifies workflow, always generates coverage, updates deps.
.github/workflows/lint_common.yml	Adds reusable pylint workflow (fail-under 9.0).
.github/workflows/functional_tests_common.yml	Refactors functional workflow, adds log artifact upload, metax toolchain compat.
.github/workflows/all_tests_metax.yml	Enables unit + functional tests for metax via common workflow.
.github/workflows/all_tests_cuda.yml	Enables unit + functional tests for cuda via common workflow.
.github/workflows/all_tests_common.yml	Orchestrates lint + unit + functional in one entry workflow.
.github/pull_request_template.md	Replaces PR template content (currently PR-specific).
.github/configs/metax.yml	Switches to BAAI runner/image and defines metax functional matrix.
.github/configs/cuda.yml	Switches to BAAI runner/image and defines expanded functional matrix.
.github/workflows/trigger-mbridge-tests.yml	Removes unused workflow.
.github/workflows/release.yaml	Removes unused workflow.
.github/workflows/release-freeze.yml	Removes unused workflow.
.github/workflows/multi-approval-bot.yml	Removes unused workflow.
.github/workflows/install-test.yml	Removes unused workflow.
.github/workflows/dependabot.yml	Removes unused workflow.
.github/workflows/copyright-check.yml	Removes unused workflow.
.github/workflows/community-bot.yml	Removes unused workflow.
.github/workflows/close-inactive-issue-pr.yml	Removes unused workflow.
.github/workflows/cherry-pick-release-commit.yml	Removes unused workflow.
.github/workflows/check_api_backwards_compatibility_workflow.yml	Removes unused workflow.
.github/workflows/auto-update-copy-pr-bot.yml	Removes unused workflow.
.github/workflows/auto-swap-labels.yml	Removes unused workflow.
.github/workflows/auto-reminder-bot.yml	Removes unused workflow.
.github/workflows/auto-assign-milestone.yml	Removes unused workflow.
.github/workflows/_update_dependencies.yml	Removes unused workflow.
.github/workflows/_release_library.yml	Removes unused workflow.
.github/workflows/_build_test_publish_wheel.yml	Removes unused workflow.
.github/copy-pr-bot.yaml	Removes unused config.
.github/CODEOWNERS	Removes CODEOWNERS file.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Darryl233 · 2026-04-17T08:34:50Z

You can disable workflows forked from the original NVIDIA Megatron-LM repository, but do not remove them like .github/workflows/_build_test_publish_wheel.yml, .github/workflows/_release_library.yml, .github/workflows/_update_dependencies.yml etc. We may need merge into NVIDIA Megatron-LM one day.

Darryl233 · 2026-04-17T09:15:06Z

Megatron-LM has its own rules; please align with the original repository.
https://github.com/NVIDIA/Megatron-LM/blob/ded22f42849e679dd65c1fba832951d7c712a3c7/.github/workflows/cicd-main.yml#L244-L285

Darryl233 · 2026-04-17T09:23:06Z

Seems not like a template to me

Darryl233 · 2026-04-17T09:24:31Z

+    try:
+        import simpy  # Guard against this import in test case
+    except ModuleNotFoundError:
+        # Fall back to enqueuing all requests immediately when simpy is unavailable.
+        return [-1] * num_requests


Why not just install simpy?

Darryl233 · 2026-04-17T09:27:44Z

Why do we add this file

Darryl233 · 2026-04-17T09:28:23Z

Why do we add this file

Darryl233 · 2026-04-17T09:31:11Z

Why do we have lots of changes in megatron/core and megatron/rl

Darryl233 · 2026-04-17T09:45:03Z

Why do we need so many code changes to fix the issues with the yq tool? Is this really necessary?

Darryl233 · 2026-04-17T09:57:59Z

Why do we add so many new config file? We only test our own cases?

CLAassistant · 2026-04-20T05:46:19Z

All committers have signed the CLA.

Darryl233 · 2026-04-21T06:25:59Z

+  - mx-4g-cicd-megatron

 # Container volumes (hardware-specific paths)
 container_volumes:


metax platform runner
container_volumes:

/mnt/op/cicd/baai_datasets:/home/gitlab-runner/data

/mnt/op/cicd/baai_tokenizers:/home/gitlab-runner/tokenizers

# Conflicts: # .github/workflows/lint_common.yml

# Conflicts: # tests/functional_tests/shell_test_utils/run_ci_test.sh

Darryl233

LGTM

lxd-cumt

LGTM

BrianPei added 4 commits April 17, 2026 11:34

fix cuda ci_image and runner_label

2237b39

Copilot AI review requested due to automatic review settings April 17, 2026 03:46

Copilot started reviewing on behalf of BrianPei April 17, 2026 03:46 View session

Copilot AI reviewed Apr 17, 2026

View reviewed changes

Comment thread .github/workflows/all_tests_common.yml

Comment thread megatron/rl/inference/megatron.py Outdated

Comment thread tests/unit_tests/test_rl_inference_megatron.py Outdated

Comment thread .github/pull_request_template.md Outdated

Comment thread .github/workflows/all_tests_common.yml

马函廷 and others added 2 commits April 17, 2026 15:42

Declare regex as base runtime dependency

9efcb3f

fix envsubst install logic

cfa308f

Darryl233 reviewed Apr 17, 2026

View reviewed changes

Comment thread .github/workflows/all_tests_common.yml

Darryl233 reviewed Apr 17, 2026

View reviewed changes

Comment thread examples/rl/environments/mock/__init__.py Outdated

Darryl233 Apr 17, 2026

Copy link
Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we add this file

Darryl233 reviewed Apr 17, 2026

View reviewed changes

Comment thread tests/functional_tests/python_test_utils/null_numeric_agent.py Outdated

Darryl233 Apr 17, 2026

Copy link
Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we add this file

Darryl233 reviewed Apr 17, 2026

View reviewed changes

BrianPei added 3 commits April 18, 2026 11:30

restore nv original ymls

31fa404

run functional_tests after unit_tests

1c17216

modify all_tests to pipeline workflow

c947ec4

Darryl233 changed the title ~~[CICD] Simplified workflows, Add lint_check & functional_tests, Switch to BAAI metax runner~~ [CICD] Simplified workflows, Add lint_check & functional_tests, Switch to FlagCICD metax runner Apr 20, 2026

Darryl233 reviewed Apr 21, 2026

View reviewed changes

Comment thread .github/configs/cuda.yml

Darryl233 reviewed Apr 21, 2026

View reviewed changes

Comment thread .github/configs/cuda.yml

Darryl233 reviewed Apr 21, 2026

View reviewed changes

AlexMa616 added 3 commits April 22, 2026 15:28

Simplify yq handling for functional tests

1fc170a

Install simpy and move mock RL agent under tests

fa607fb

Unify optimizer param group key handling

da46542

AlexMa616 added 17 commits April 22, 2026 15:28

Revert unrelated Megatron source changes

74ce8e7

Shrink functional coverage to minimal reviewer scope

c20508d

Temporarily disable unit tests to verify functional flow

9524d3a

Fix functional job matrix in cuda workflow

98191cf

Avoid skipping functional job when unit tests are disabled

b32b657

Add no-shared-fs for minimal GPT functional case

434af08

Restore reviewer runner config and unit flow on pr-0417

72371c6

add metax BAAI container_volumes

e4315f5

update pr_template

cd9aba9

Limit lint setup to lint tools only

d6cfe90

# Conflicts: # .github/workflows/lint_common.yml

Use isolated tool env for lint workflow

f435415

Fix root path initialization in functional test runner

dc9aedb

# Conflicts: # tests/functional_tests/shell_test_utils/run_ci_test.sh

use original lint logic

cf10b49

lint add install uv

a91eb1b

Remove invalid no-shared-fs arg from minimal GPT case

907063a

debug lint; metax add ignore unitcase

24176b7

Handle missing no-shared-fs arg in plugin utils

3e764ad

AlexMa616 force-pushed the pr-0417 branch from f9f67f9 to 3e764ad Compare April 22, 2026 07:30

AlexMa616 and others added 7 commits April 22, 2026 16:24

Restore upstream lint flow with minimal bootstrap

b03a04f

Add qwen3 metax functional regression case

7d844ca

Tune qwen3 functional case for A100 validation

a16b4ef

add log-memory params for unit_test_core

75ea7f7

Use platform-specific qwen3 golden for metax

7604756

unittests install torch; qwen case add log params

3c97648

Update qwen3 c500 golden baseline

2da1cef

Darryl233 approved these changes Apr 24, 2026

View reviewed changes

keep lint trigger when PR

846995f

lxd-cumt approved these changes Apr 24, 2026

View reviewed changes

xmhubj approved these changes Apr 24, 2026

View reviewed changes

Darryl233 merged commit 89105ae into flagos-ai:main Apr 24, 2026
85 of 102 checks passed

Uh oh!

Conversation

BrianPei commented Apr 17, 2026

Description

Type of change

Changes

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Darryl233 commented Apr 17, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Darryl233 commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CLAassistant commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Darryl233 Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Darryl233 left a comment

Choose a reason for hiding this comment

Uh oh!

lxd-cumt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Darryl233 commented Apr 17, 2026 •

edited

Loading

CLAassistant commented Apr 20, 2026 •

edited

Loading

Darryl233 Apr 21, 2026 •

edited

Loading