Add Halo LFM2.5 Expert Parallel and vision training notebooks - #120
Add Halo LFM2.5 Expert Parallel and vision training notebooks#120advpropsys wants to merge 1 commit into
Conversation
|
hey @iamleonie pinging you as discussed re Halo partnership 🚀 |
iamleonie
left a comment
iamleonie
left a comment
There was a problem hiding this comment.
Thank you for the contributions.
I have added a few comments to match the tutorial style to our existing tutorials and help developers new to these topics get a better understanding.
There was a problem hiding this comment.
can we please rename this to "sft_moe_with_halo.ipynb" to match naming convention of our fine-tuning notebooks?
There was a problem hiding this comment.
can we please rename this to "sft_moe_for_vision_language_model_with_halo.ipynb" to match naming convention of our fine-tuning notebooks?
| | Name | Framework | Description | Link | | ||
| |------|-----------|-------------|------| | ||
| | **Text Models** | | | | | ||
| | Mixture-of-Experts SFT | [Halo](https://github.com/whitecircle/halo) | Fine-tune LFM2.5-8B-A1B with two-way expert parallelism | [Notebook](./finetuning/notebooks/lfm25_8b_a1b_with_halo.ipynb) | |
There was a problem hiding this comment.
can you please move this below the cell for SFT with TRL with the following edits:
- Name: "Supervised Fine-Tuning (SFT) for Mixture-of-Experts"
- Framework: please also add link to your docs like we have for the other examples
- Description: "Supervised- fine-tuning of LFM2.5-8B-A1B Mixture-of-Expert with two-way expert parallellism using Halo"
| | **Vision Models** | | | | | ||
| | Supervised Fine-Tuning (SFT) | [Unsloth](https://github.com/unslothai/unsloth) ([Docs](https://docs.liquid.ai/lfm/fine-tuning/unsloth)) | Supervised fine-tuning for LFM2-VL models on custom image-text datasets | [Notebook](./finetuning/notebooks/sft_for_vision_language_model.ipynb) [](https://colab.research.google.com/github/Liquid4All/cookbook/blob/main/finetuning/notebooks/sft_for_vision_language_model.ipynb) | | ||
| | Supervised Fine-Tuning (SFT) | [TRL](https://github.com/huggingface/trl) ([Docs](https://docs.liquid.ai/lfm/fine-tuning/trl)) | Supervised fine-tuning for LFM2.5-VL models using TRL with parameter-efficient LoRA | [Notebook](./finetuning/notebooks/sft_for_vision_language_model_with_trl.ipynb) [](https://colab.research.google.com/github/Liquid4All/cookbook/blob/main/finetuning/notebooks/sft_for_vision_language_model_with_trl.ipynb) | | ||
| | Object detection LoRA benchmark | [Halo](https://github.com/whitecircle/halo) | Fine-tune LFM2.5-VL-3B on CPPE-5 and measure held-out detection F1 | [Notebook](./finetuning/notebooks/lfm25_vl_object_detection_with_halo.ipynb) | |
There was a problem hiding this comment.
Can you please make the following edits:
- rename the "Name" to " Supervised Fine-tuning (SFT)
- please add link to the docs
- Description: "Supervised fine-tuning for LFM2.5-VL-3B on CPPE-5 for object detection"
| "cell_type": "markdown", | ||
| "id": "intro", | ||
| "metadata": {}, | ||
| "source": "# Fine-tune LFM2.5-8B-A1B with Halo\n\nThis notebook fine-tunes [LFM2.5-8B-A1B](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B) on UltraChat 200K.\n\nHalo distributes the model's 32 routed experts across two GPUs. Each process owns 16 experts during training.\n\nHalo gathers the expert weights when it saves. The result is a standard Hugging Face checkpoint.\n" |
There was a problem hiding this comment.
As we're not able to run this notebook in a Google Colab like our other notebooks, I think it would be helpful to developers if we could get a short "Prerequisites" section for what kind of GPU is needed (you mention two matching NVIDIA GPUs and large data volume later, but I think having it as overview would be helpful)
Maybe a note like this could be helpful as well.
Where to run: These notebooks require Hopper or Blackwell GPUs (H100, H200, B200, B300) with Docker. They do not run on Google Colab. Cloud GPU providers such as Lambda Labs, RunPod, or Vast.ai offer compatible instances.
I think it could also be helpful to give the readers a rough estimate of how much training time to anticipate for at least one GPU type (e.g. X hours for two H100s)
| "id": "tuned-title", | ||
| "metadata": {}, | ||
| "source": "## Measure the fine-tuned model\n" | ||
| }, |
There was a problem hiding this comment.
It'd be nice to show one inference example here in a section called "Inference"
| "cell_type": "markdown", | ||
| "id": "load-title", | ||
| "metadata": {}, | ||
| "source": "## Load the saved checkpoint\n\nThe saved checkpoint uses the native Transformers format.\n" |
There was a problem hiding this comment.
can we call this section "Inference" please to match our other notebooks
| { | ||
| "cell_type": "markdown", | ||
| "id": "runtime", | ||
| "metadata": {}, |
There was a problem hiding this comment.
can we get some explanations here for a few of the settings:
for example:
- CUDA_DEVICE_MAX_CONNECTIONS=1 -> why is this set here and not in the VL notebook? does it have something to do with the multi GPU setup? could be explained briefly.
- why we use --network host here and -p 8888:8888 in the VL notebook?
| "cell_type": "markdown", | ||
| "id": "dry-run-title", | ||
| "metadata": {}, | ||
| "source": "## Check the launch command\n\nThe dry run prints the two-process command without loading the model.\n" |
There was a problem hiding this comment.
Should we have this dry run check in the VL notebook as well?
| "id": "config", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": "CONFIG_PATH.write_text(\n r\"\"\"\nmodel_name_or_path: LiquidAI/LFM2.5-VL-3B\ntrust_remote_code: true\nattn_implementation: sdpa\n\ndataset:\n- /mnt/lfm25-cppe5/cppe5-halo\nconversation_field: messages\nimages_field: image\ntest_size: null\n\ntrain_on_completions_only: true\nassistant_message_template: \"<|im_start|>assistant\\n\"\nmax_length: 4096\npacking: false\npadding_free: false\n\nuse_peft: true\nlora_r: 16\nlora_alpha: 32\nlora_dropout: 0.05\nlora_target_modules:\n- q_proj\n- k_proj\n- v_proj\n- out_proj\n- in_proj\n- w1\n- w2\n- w3\n- linear_1\n- linear_2\n- fc1\n- fc2\nlora_task_type: CAUSAL_LM\n\nper_device_train_batch_size: 1\nper_device_eval_batch_size: 1\ngradient_accumulation_steps: 8\nmax_steps: 150\ngradient_checkpointing: true\ngradient_checkpointing_kwargs:\n use_reentrant: false\noptim: adamw_torch_fused\nlearning_rate: 2.0e-05\nweight_decay: 0.01\nlr_scheduler_type: cosine\nwarmup_steps: 10\nmax_grad_norm: 1.0\nbf16: true\nseed: 42\n\noutput_dir: /mnt/lfm25-cppe5/lfm25-vl-3b-cppe5-halo\nsave_strategy: steps\nsave_steps: 150\neval_strategy: \"no\"\nsave_total_limit: 1\nsave_only_model: true\nlogging_steps: 1\nlogging_first_step: true\nreport_to: none\n\ndataloader_num_workers: 0\ndataset_num_proc: 1\nremove_unused_columns: false\n\"\"\".lstrip()\n)\nprint(CONFIG_PATH)\n" |
There was a problem hiding this comment.
param in MoE notebook is called "train_only_on_completions" vs here "train_on_completions_only". is this a typo, alias, or genuinely different params? worth clarifying.
c92d93e to
5a03eab
Compare
2 participants
Summary
LFM Expert Parallelism
Halo is the first training framework to support Expert Parallelism for the LFM model family.
The notebook distributes LFM2.5-8B-A1B's 32 routed experts across two GPUs. Each process owns 16 experts during training.
Halo gathers the expert weights when it saves the model. The output remains a standard Hugging Face checkpoint.
This uses the native Hugging Face model implementation. It does not require a model fork or checkpoint conversion.
Runtime
Both notebooks detect Hopper or Blackwell GPUs and select the matching public Halo image. There is no tested support for Ampere/Tesla or older generation.
Validation
public.ecr.aws/whitecircle/halo:blackwell.AI assistance
Codex with GPT-5.6 assisted with the notebook implementation and validation.