Skip to content

[Feature Request] MOBA:Mixture of Block Attention For Long-context #3850

@fengxy-03

Description

@fengxy-03

Summary
Request implementation of MoBA Attention, the technique used by Kimi for efficient long-context training and inference.

Current state
Kimi official has already placed the basic code in their repository. Are there any plans to integrate it into megatron-lm?
MoBA:https://github.com/MoonshotAI/MoBA

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions