[sglang] Fix megatron support in sglang and add sglang_async support & CI tasks

[SwordFaith]

Checklist Before Starting

• Search for similar PR(s).

What does this PR do?

Add one-line overview of what this PR aims to achieve or accomplish.

• Fix sglang megatron support
• Add sglang_async megatron support
• Add CI task to protect megatron-sglang impl

Test

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluatuion results, etc.

https://wandb.ai/swordfaith/gsm8k_async_rl/runs/6h7apmbn?nw=nwuserswordfaith

Additional Info.

• Issue Number: Fixes issue # or discussion # if any.
• Training: [Note which backend this PR will affect: FSDP, Megatron, both, or none]
• Inference: SGLang

Checklist Before Submitting

• Read the Contribute Guide.
• Apply pre-commit checks.
• Add [BREAKING] to the PR title if it breaks any API.
• Update the documentation about your changes in the docs.
• Add CI test(s) if necessary.

[ETOgaosion]

Great work! Thanks a lot for your efforts to help support sglang and megatron to catch up with vLLM.

[ETOgaosion]

Now this CI file has become so complicate that is hard to maintain both vLLM and SGLang tests. My suggestion is that we can add both vLLM and SGLang tests in a single shell script test file https://github.com/volcengine/verl/blob/main/tests/e2e/run_ppo_trainer_megatron.sh, and we simply need to call this file once.

[ETOgaosion]

I mean here we can also include SGLang python scripts below the vLLM, so that we can test two systems with the same configuration.

[SwordFaith]

OK, I will modify it configuration after CI test finished

[SwordFaith]

Ok, I'll modify this.

[ETOgaosion]

I read somewhere that the SGLang's TP rank output might be different from vLLM's, as now the dispatch method is the same with FSDP, can we just borrow the fsdp_sglang.py's implementation to avoid misalignment?
In here

[SwordFaith]

It seems that we currently lack a unified weight conversion method for FSDP as Megatron Core, so we cannot merge these two classes for now. Are there any better approaches to implement this?

[ETOgaosion]

Oh, I mean not merging these classes, we can borrow the implementation there, just like this vLLM's implementation

[ETOgaosion]

@SwordFaith Thanks for contribution, could you rebase main and add some megatron_sglang support of training side expert parallel as #1467 ? And you can use the latest image to add some tests~

[SwordFaith]

@SwordFaith Thanks for contribution, could you rebase main and add some megatron_sglang support of training side expert parallel as #1467 ? And you can use the latest image to add some tests~

For sglang, it seems that support has been added in utils/megatron_utils as part of #1467 . It would be better to share the megatron_utils per_tensor_generator implementation between megatron vllm and sglang. However, I believe such a refactor would require end-to-end training verification, which could take days to modify and debug. For now, we can address sglang support in the current PR and plan the refactor for the future.

[vermouth1992]

Shall we merge this? @ETOgaosion

[ETOgaosion]

Yes, finally, and we can test larger models based on SGLang backend.