[Ascend] Add vendor:ascend backend#43
Conversation
|
|
|
|
||
| impls = [ | ||
| # Activations - Forward | ||
| OpImpl(op_name="gelu", impl_id="vendor.ascend", kind=BackendImplKind.VENDOR, fn=_bind_is_available(backend.gelu, is_avail), vendor="ASCEND", priority=100), |
There was a problem hiding this comment.
Why was only one gelu function registered?
During the Qwen3 adaptation process, can all operators run successfully based on the FlagOS backend? Is there no dependency on the Vendor/ascend backend?
There was a problem hiding this comment.
Only by adding the gelu registration under vendor and other initialization modules, the In-tree test case (rmsnorm_fwd) provided in PR#4 was successfully run, and the test showed that the adaptation code of TE-FL on ascend functions normally.
During the execution of Qwen3, according to the printed log "TE-FL manager.py:417 INFO] Op 'multi_tensor_adam' using 'default.flagos' (kind=flagos, vendor=None)", it was confirmed that multi_tensor_adam was successfully replaced. There were no printed logs for other operators, indicating that vendor dependencies are not involved at this time.
There was a problem hiding this comment.
Does this mean that when training Qwen3 on ASCEND, most of the ops in transformer_engine were not used, and only multi_tensor_adam was utilized?
There was a problem hiding this comment.
Perhaps this is so.
When adapting to TE on NPU, MindSpeed replaces some implementations, which may be the reason for the impact.
There was a problem hiding this comment.
Could you provide a YAML file for training Qwen3 with FlagScale? We'd like to confirm the system configuration.
Please complete the PR description |
done |
|
Please pull the latest main branch, and format code, and pass CICD tests |
3 participants
Description
Add vendor:ascend backend.Please review.
Fixes # (issue)
Type of change
Changes
Please list the changes introduced in this PR:
Checklist: