Skip to content

v3.12.0

Choose a tag to compare

View all tags
@Jintao-Huang Jintao-Huang released this 30 Dec 03:24
· 268 commits to main since this release
v3.12.0
5effa64

中文版

新特性

  1. Megatron-SWIFT
    a. GKD算法支持Megatron训练，文档参考：https://swift.readthedocs.io/zh-cn/latest/Megatron-SWIFT/GKD.html
    b. 新模型支持：GLM4 Dense; GLM4.7; GLM4.6v-Flash, GLM-4.1V。
    c. save_safetensors 支持断点续训，将 Mcore-Bridge 加载和存储方式作为推荐方式。
    d. 非 padding-free 训练模式支持更多训练阶段：GRPO/DPO/KTO/RM/序列分类。
    e. group_by_length 参数支持，将数据集长度大致相同的样本分组在一起（含随机因素），加速非packing模式下训练速度。
    f. 支持 --report_to 参数，将训练日志在 wandb/swanlab 中记录并可视化。
    g. Qwen3-Next 使用 Zero-Centered RMSNorm，与 transformers 对齐。
    h. train_dataloader_shuffle 参数支持，控制训练数据集是否随机。
    i. template.encode 新增重试机制，避免 megatron 训练因网络问题获取图片/视频报错而卡住。
  2. RL
    a. 增加 Off-Policy Sequence Masking (from DeepSeek-V3.2)，文档参考：https://swift.readthedocs.io/zh-cn/latest/Instruction/GRPO/AdvancedResearch/training_inference_mismatch.html#off-policy-sequence-masking
    b. GRPO 增加参数 num_generations_eval 设置 eval 阶段的生成数量。
    c. 优化 GKD loss 计算的显存峰值。
    d. GRPO/GKD server mode 支持使用 ipv6 地址。
    e. 支持使用 structured_outputs_regex 进行结构化输出采样。
  3. 训练
    a. embedding/reranker/序列分类任务支持序列 packing 和序列并行。训练脚本参考：https://github.com/modelscope/ms-swift/tree/main/examples/train/sequence_parallel
    b. 支持 --fsdp fsdp2 使用 ms-swift 内置的 FSDP2 配置文件。
    c. loss_scale 支持3种基本策略：'default'、'last_round'、'all'与其他策略的混合使用，例如：'last_round+ignore_empty_think'。
    d. cached_dataset 支持 embedding/reranker/序列分类训练任务，训练脚本参考https://github.com/modelscope/ms-swift/tree/main/examples/train/cached_dataset
    e. thinking template 重构，ThinkingTemplate 功能合入 Template，新增enable_thinking, add_non_thinking_prefix参数。
    f. 新增 SWIFT_PATCH_CONV3D 环境变量，避免 torch2.9 环境跑 conv3d 缓慢的问题。
    g. 支持 swanlab_notification_method 参数，在训练完成/发生错误时，指定 swanlab 通知方式。
    h. dataloader_prefetch_factor 参数默认值从10修改为2。
  4. 国产化硬件（感谢昇腾和招商银行技术团队的贡献）
    a. 新增更多训练脚本：https://github.com/modelscope/ms-swift/tree/main/examples/ascend
    b. Qwen3-VL 混合算子支持，具体查看这个PR：#7079
    c. 更新 Megatron-SWIFT NPU 性能采集/精度采集相关文档，参考这里：https://swift.readthedocs.io/zh-cn/latest/Megatron-SWIFT/Ascend.html

新模型

  1. 纯文本模型：
    a. ZhipuAI/GLM-4.7系列
    b. iic/QwenLong-L1.5-30B-A3B
    c. gongjy/MiniMind2 （感谢 @PiggerZZM 的贡献）
  2. 多模态模型：
    a. ZhipuAI/GLM-4.6V; ZhipuAI/GLM-4.6V-Flash系列
    b. Tencent-Hunyuan/HunyuanOCR

English Version

New Features

  1. Megatron-SWIFT
    a. GKD algorithm supports Megatron training. Documentation reference: https://swift.readthedocs.io/en/latest/Megatron-SWIFT/GKD.html
    b. New model support: GLM4 Dense; GLM4.7; GLM4.6v-Flash, GLM-4.1V.
    c. save_safetensors supports checkpoint resumption, with Mcore-Bridge loading and storage method as the recommended approach.
    d. Non-padding-free training mode supports more training stages: GRPO/DPO/KTO/RM/sequence classification.
    e. group_by_length parameter support, grouping samples with similar lengths in the dataset together (with random factors) to accelerate training speed in non-packing mode.
    f. Support for --report_to parameter to log and visualize training logs in wandb/swanlab.
    g. Qwen3-Next uses Zero-Centered RMSNorm, aligned with transformers.
    h. train_dataloader_shuffle parameter support to control whether training dataset is shuffled.
    i. Added retry mechanism to template.encode to prevent megatron training from freezing due to network issues when fetching images/videos.
  2. RL
    a. Added Off-Policy Sequence Masking (from DeepSeek-V3.2). Documentation reference: https://swift.readthedocs.io/en/latest/Instruction/GRPO/AdvancedResearch/training_inference_mismatch.html#off-policy-sequence-masking
    b. GRPO adds num_generations_eval parameter to set the number of generations during eval stage.
    c. Optimized memory peak for GKD loss calculation.
    d. GRPO/GKD server mode supports using ipv6 addresses.
    e. Support for structured output sampling using structured_outputs_regex.
  3. Training
    a. Embedding/reranker/sequence classification tasks support sequence packing and sequence parallelism. Training script reference: https://github.com/modelscope/ms-swift/tree/main/examples/train/sequence_parallel
    b. Support for --fsdp fsdp2 to use ms-swift built-in FSDP2 configuration file.
    c. loss_scale supports 3 basic strategies: 'default', 'last_round', 'all' and their hybrid use with other strategies, e.g., 'last_round+ignore_empty_think'.
    d. cached_dataset supports embedding/reranker/sequence classification training tasks. Training script reference: https://github.com/modelscope/ms-swift/tree/main/examples/train/cached_dataset
    e. Thinking template refactored, ThinkingTemplate functionality merged into Template, added enable_thinking and add_non_thinking_prefix parameters.
    f. Added SWIFT_PATCH_CONV3D environment variable to avoid slow conv3d execution in torch2.9 environment.
    g. Support for swanlab_notification_method parameter to specify swanlab notification method when training completes/errors occur.
    h. dataloader_prefetch_factor parameter default value changed from 10 to 2.
  4. Domestic Hardware (Thanks to Ascend and CMB technical teams)
    a. Added more training scripts: https://github.com/modelscope/ms-swift/tree/main/examples/ascend
    b. Qwen3-VL hybrid operator support, see this PR: #7079
    c. Updated Megatron-SWIFT NPU performance collection/accuracy collection documentation, reference: https://swift.readthedocs.io/en/latest/Megatron-SWIFT/Ascend.html

New Models

  1. Text-only models:
    a. ZhipuAI/GLM-4.7 series
    b. iic/QwenLong-L1.5-30B-A3B
    c. gongjy/MiniMind2 (Thanks to @PiggerZZM's contribution)
  2. Multimodal models:
    a. ZhipuAI/GLM-4.6V; ZhipuAI/GLM-4.6V-Flash series
    b. Tencent-Hunyuan/HunyuanOCR

What's Changed

New Contributors

Full Changelog: v3.11.0...v3.12.0

Contributors

liuyanyi, PiggerZZM, and 12 other contributors
Assets 2
Loading