Summary
Request implementation of MoBA Attention, the technique used by Kimi for efficient long-context training and inference.
Current state
Kimi official has already placed the basic code in their repository. Are there any plans to integrate it into megatron-lm?
MoBA:https://github.com/MoonshotAI/MoBA