diff --git a/README.md b/README.md index b35461f5bf..c2cde635d6 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,18 @@ NVIDIA BioNeMo Framework is a comprehensive suite of programming tools, librarie Training benchmarks for ESM-2, a well known protein sequence model using the BERT architecture.

+

+ +
+ ESM-2 low-precision training benchmarks (BF16 vs MXFP8 vs NVFP4) on NVIDIA B300 SXM GPUs with FSDP2. +

+ +

+ +
+ Llama 3 70B context parallelism scaling on 36x NVIDIA GB300 NVL36, from 8K to 144K context length. +

+ ## ⚡ Quick Start ```bash @@ -43,6 +55,11 @@ cd bionemo-framework/bionemo-recipes/recipes/esm2_native_te/ ## Recent News +- 03/09/2026 [Qwen2.5 / Qwen3 model](bionemo-recipes/models/qwen/) with TE acceleration, FP8/MXFP8, KV-cache inference, and bidirectional HF checkpoint conversion. +- 03/05/2026 [ESM2 NVFP4 and MXFP8](bionemo-recipes/recipes/esm2_native_te/README.md#low-precision-performance-benchmarks) low-precision training — up to **2,367 TFLOPS/GPU** on NVIDIA B300 at 15B scale with per-layer precision control. +- 02/23/2026 [Mixtral MoE model](bionemo-recipes/models/mixtral/) with TE `GroupedLinear` for efficient parallel expert computation, FP8/FP4 support, and HF conversion. +- 02/13/2026 [ESM2 PEFT recipe](bionemo-recipes/recipes/esm2_peft_te/) for LoRA fine-tuning with sequence packing support. +- 01/14/2026 [Llama3 Context Parallelism](bionemo-recipes/recipes/llama3_native_te/README.md#performance-benchmarks) — scaling Llama 3 70B to 144K context on 36x GB300 NVL36 with ~65% MFU. - 10/27/2025 [CodonFM recipe](https://github.com/NVIDIA/bionemo-framework/tree/main/bionemo-recipes/recipes/codonfm_ptl_te) released! This is an accelerated version of the original [research codebase](https://github.com/NVIDIA-Digital-Bio/CodonFM) with [scientific preprint](https://research.nvidia.com/labs/dbr/assets/data/manuscripts/nv-codonfm-preprint.pdf). - 09/30/2025 Megatron/NeMo 5D parallel BioNeMo Framework image v2.7 [released on NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/containers/bionemo-framework) for both x86 and ARM CPUs. - 09/01/2025 [bionemo-recipes](https://github.com/NVIDIA/bionemo-framework/tree/main/bionemo-recipes) goes live! Lightweight and portable examples with state-of-the-art training performance you can riff on to meet your needs. @@ -61,13 +78,18 @@ A core use-case of the BioNeMo Framework is to help digital biology scientists a | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------- | ----------- | ------------- | ------ | ---------------- | ------ | ------------------- | | `models/`
`amplify` | TE accelerated protein BERT, pushed to HuggingFace | ✅ Active | ❌ | ✅ | ✅ | 🚧 WIP | ✅ | 🚧 WIP | | `models/`
`esm2` | TE accelerated protein BERT, pushed to HuggingFace | ✅ Active | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | -| `models/`
`llama3` | TE accelerated Llama 3 | ✅ Active | ❌ | 🚧 WIP | ✅ | ✅ | 🚧 WIP | 🚧 WIP | +| `models/`
`llama3` | TE accelerated Llama 3 | ✅ Active | ❌ | 🚧 WIP | ✅ | ✅ | ✅ | ✅ | | `models/`
`geneformer` | TE accelerated single-cell BERT | 🚧 WIP | ❌ | ✅ | 🚧 WIP | 🚧 WIP | 🚧 WIP | 🚧 WIP | | `recipes/`
`codonfm_ptl_te` | Recipe for [CodonFM](https://research.nvidia.com/labs/dbr/assets/data/manuscripts/nv-codonfm-preprint.pdf)'s Encodon using TE | ✅ Active | ❌ | 🚧 WIP | ✅ | ✅ | 🚧 WIP | 🚧 WIP | | `recipes/`
`esm2_accelerate_te` | Recipe for ESM2 TE + HF Accelerate | ✅ Active | ❌ | 🚧 WIP | ✅ | ❌ | ✅ | 🚧 WIP | -| `recipes/`
`esm2_native_te` | Recipe for ESM2 TE + native PyTorch | ✅ Active | ❌ | ✅ | ✅ | ✅ | ✅ | 🚧 WIP | +| `recipes/`
`esm2_native_te` | Recipe for ESM2 TE + native PyTorch | ✅ Active | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | | `recipes/`
`geneformer_native_te_mfsdp_fp8` | Recipe for Geneformer HF model | 🚧 WIP | ❌ | ✅ | ✅ | ❌ | ✅ | 🚧 WIP | -| `recipes/`
`llama3_native_te` | Recipe for Llama 3 TE + native PyTorch | ✅ Active | ❌ | 🚧 WIP | ✅ | ✅ | 🚧 WIP | 🚧 WIP | +| `recipes/`
`llama3_native_te` | Recipe for Llama 3 TE + native PyTorch | ✅ Active | ❌ | 🚧 WIP | ✅ | ✅ | ✅ | ✅ | +| `models/`
`mixtral` | TE accelerated MoE model | ✅ Active | ❌ | 🚧 WIP | ✅ | ✅ | ✅ | 🚧 WIP | +| `models/`
`qwen` | TE accelerated Qwen2.5/Qwen3 | ✅ Active | ❌ | 🚧 WIP | ✅ | ✅ | ✅ | 🚧 WIP | +| `recipes/`
`esm2_peft_te` | Recipe for ESM2 LoRA fine-tuning | ✅ Active | ❌ | ❌ | ✅ | ✅ | 🚧 WIP | ❌ | +| `recipes/`
`evo2_megatron` | Recipe for Evo2 via Megatron Bridge | 🚧 WIP | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | +| `recipes/`
`fp8_analysis` | FP8 training analyzer & heatmap tool | ✅ Active | N/A | N/A | N/A | N/A | N/A | N/A | | `recipes/`
`vit` | Recipe for Vision Transformer | 🚧 WIP | ❌ | ✅ | ✅ | ❌ | ✅ | 🚧 WIP | @@ -113,7 +135,7 @@ BioNeMo Framework is part of a larger ecosystem of NVIDIA Biopharma products. Ge ## Documentation Resources -- **Official Documentation:** Contents of `sub-packages` including user guides, API references, and troubleshooting, are documented on our [official documentation](https://docs.nvidia.com/bionemo-framework/latest/). Nightly builds of this documentation is available on [BioNeMo Framework GitHub Pages](https://nvidia.github.io/bionemo-framework/) +- **Official Documentation:** Documentation for sub-packages, including user guides, API references, and troubleshooting, is available on our [official documentation](https://docs.nvidia.com/bionemo-framework/latest/). Nightly builds of this documentation is available on [BioNeMo Framework GitHub Pages](https://nvidia.github.io/bionemo-framework/) - **🚧 In-Progress Documentation 🚧:** `bionemo-recipes` documentation is currently work in progress, however the recipes are meant to be self-documented and easy to understand—we suggest you throw them into your favorite genai code assistant! @@ -136,8 +158,8 @@ docker run --rm -it \ #### Initializing 3rd-party dependencies as git submodules -The NeMo and Megatron-LM dependencies are included as git submodules in bionemo2. The pinned commits for these submodules represent the "last-known-good" versions of these packages -that are confirmed to be working with bionemo2 (and those that are tested in CI). +The NeMo and Megatron-LM dependencies are included as git submodules in BioNeMo Framework. The pinned commits for these submodules represent the "last-known-good" versions of these packages +that are confirmed to be working with BioNeMo Framework (and those that are tested in CI). To initialize these sub-modules when cloning the repo, add the `--recursive` flag to the git clone command: diff --git a/bionemo-recipes/models/amplify/README.md b/bionemo-recipes/models/amplify/README.md index a1f7e8fe70..0025d7b73c 100644 --- a/bionemo-recipes/models/amplify/README.md +++ b/bionemo-recipes/models/amplify/README.md @@ -117,5 +117,3 @@ Or, upload all models at once with: ```bash for dir in *; do huggingface-cli upload nvidia/$(basename "$dir") "$dir/"; done ``` - -z