Fine-tuning hardware resource issues

[gml-blip]

Hello, how many H100s and how much video memory are needed to fine-tune the evo2-7b model?

[trvachov]

Here is some data that might help you.

Run command:

torchrun --nproc-per-node 4 --nnodes 1 /usr/local/bin/train_evo2 -d ./sub-packages/bionemo-evo2/examples/training_data_config.yaml  --dataset-dir ./preprocessed_data --result-dir pretraining_demo --experiment-name evo2 --model-size 7b --devices 4 --num-nodes 1 --seq-length 8192 --micro-batch-size 1 --lr 0.000015 --min-lr 0.0000149 --warmup-steps 100 --grad-acc-batches 4 --max-steps 100  --clip-grad 250 --wd 0.001 --attention-dropout 0.01 --hidden-dropout 0.01 --val-check-interval 50  --create-tensorboard-logger --ckpt-async-save

The data was taken by pulling from the notebook ./sub-packages/bionemo-evo2/examples/fine-tuning-tutorial.ipynb

Some relevant information from the logs:

┏━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃   ┃ Name                                ┃ Type                   ┃ Params ┃ Mode  ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ 0 │ module                              │ DDP                    │  6.5 B │ train │
│ 1 │ module.module                       │ Float16Module          │  6.5 B │ train │
│ 2 │ module.module.module                │ HyenaModel             │  6.5 B │ train │
│ 3 │ module.module.module.embedding      │ LanguageModelEmbedding │  2.1 M │ train │
│ 4 │ module.module.module.rotary_pos_emb │ RotaryEmbedding        │      0 │ train │
│ 5 │ module.module.module.decoder        │ HyenaStack             │  6.5 B │ train │
│ 6 │ module.module.module.output_layer   │ ColumnParallelLinear   │      0 │ train │
└───┴─────────────────────────────────────┴────────────────────────┴────────┴───────┘
Trainable params: 6.5 B
Non-trainable params: 0
Total params: 6.5 B
Total estimated model params size (MB): 25.9 K
Modules in train mode: 452
Modules in eval mode: 0

...

Training epoch 0, iteration 32/99 | lr: 4.8e-06 | global_batch_size: 16 | global_step: 32 | reduced_train_loss: 1.436 | train_step_timing in s: 4.929 | consumed_samples: 528
Training epoch 0, iteration 33/99 | lr: 4.95e-06 | global_batch_size: 16 | global_step: 33 | reduced_train_loss: 1.483 | train_step_timing in s: 4.938 | consumed_samples: 544
Training epoch 0, iteration 34/99 | lr: 5.1e-06 | global_batch_size: 16 | global_step: 34 | reduced_train_loss: 1.58 | train_step_timing in s: 4.93 | consumed_samples: 560

nvidia-smi output

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.216.03             Driver Version: 535.216.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA H100 80GB HBM3          On  | 00000000:19:00.0 Off |                    0 |
| N/A   68C    P0             689W / 700W |  66809MiB / 81559MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          On  | 00000000:2D:00.0 Off |                    0 |
| N/A   60C    P0             686W / 700W |  66861MiB / 81559MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA H100 80GB HBM3          On  | 00000000:3F:00.0 Off |                    0 |
| N/A   59C    P0             691W / 700W |  66595MiB / 81559MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA H100 80GB HBM3          On  | 00000000:66:00.0 Off |                    0 |
| N/A   54C    P0             688W / 700W |  66595MiB / 81559MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

@gml-blip does this help? (apologies for the delayed reply).

[gml-blip]

Thank you, that's very helpful. Could you please provide the code for train_evo2

Here is some data that might help you.

Run command:

torchrun --nproc-per-node 4 --nnodes 1 /usr/local/bin/train_evo2 -d ./sub-packages/bionemo-evo2/examples/training_data_config.yaml  --dataset-dir ./preprocessed_data --result-dir pretraining_demo --experiment-name evo2 --model-size 7b --devices 4 --num-nodes 1 --seq-length 8192 --micro-batch-size 1 --lr 0.000015 --min-lr 0.0000149 --warmup-steps 100 --grad-acc-batches 4 --max-steps 100  --clip-grad 250 --wd 0.001 --attention-dropout 0.01 --hidden-dropout 0.01 --val-check-interval 50  --create-tensorboard-logger --ckpt-async-save

The data was taken by pulling from the notebook ./sub-packages/bionemo-evo2/examples/fine-tuning-tutorial.ipynb

Some relevant information from the logs:

┏━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃   ┃ Name                                ┃ Type                   ┃ Params ┃ Mode  ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ 0 │ module                              │ DDP                    │  6.5 B │ train │
│ 1 │ module.module                       │ Float16Module          │  6.5 B │ train │
│ 2 │ module.module.module                │ HyenaModel             │  6.5 B │ train │
│ 3 │ module.module.module.embedding      │ LanguageModelEmbedding │  2.1 M │ train │
│ 4 │ module.module.module.rotary_pos_emb │ RotaryEmbedding        │      0 │ train │
│ 5 │ module.module.module.decoder        │ HyenaStack             │  6.5 B │ train │
│ 6 │ module.module.module.output_layer   │ ColumnParallelLinear   │      0 │ train │
└───┴─────────────────────────────────────┴────────────────────────┴────────┴───────┘
Trainable params: 6.5 B
Non-trainable params: 0
Total params: 6.5 B
Total estimated model params size (MB): 25.9 K
Modules in train mode: 452
Modules in eval mode: 0

...

Training epoch 0, iteration 32/99 | lr: 4.8e-06 | global_batch_size: 16 | global_step: 32 | reduced_train_loss: 1.436 | train_step_timing in s: 4.929 | consumed_samples: 528
Training epoch 0, iteration 33/99 | lr: 4.95e-06 | global_batch_size: 16 | global_step: 33 | reduced_train_loss: 1.483 | train_step_timing in s: 4.938 | consumed_samples: 544
Training epoch 0, iteration 34/99 | lr: 5.1e-06 | global_batch_size: 16 | global_step: 34 | reduced_train_loss: 1.58 | train_step_timing in s: 4.93 | consumed_samples: 560

nvidia-smi output

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.216.03             Driver Version: 535.216.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA H100 80GB HBM3          On  | 00000000:19:00.0 Off |                    0 |
| N/A   68C    P0             689W / 700W |  66809MiB / 81559MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          On  | 00000000:2D:00.0 Off |                    0 |
| N/A   60C    P0             686W / 700W |  66861MiB / 81559MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA H100 80GB HBM3          On  | 00000000:3F:00.0 Off |                    0 |
| N/A   59C    P0             691W / 700W |  66595MiB / 81559MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA H100 80GB HBM3          On  | 00000000:66:00.0 Off |                    0 |
| N/A   54C    P0             688W / 700W |  66595MiB / 81559MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

@gml-blip does this help? (apologies for the delayed reply).

Thank you, that's very helpful. Could you please provide the code for train_evo2

[igoralves1]

what is the recommended hardware to run bionemo?

[trvachov]

Thanks for the questions everyone!

To summarize the answers and point to relevant resources:

Hardware for fine-tuning evo2-7b: As shared above, fine-tuning the 7b model was tested with 4x NVIDIA H100 80GB HBM3 GPUs, using ~66.5 GB VRAM per GPU with micro-batch-size 1 and seq-length 8192.

train_evo2 source code: This is an installed entry point from the bionemo-evo2 package. The source code is at sub-packages/bionemo-evo2/src/bionemo/evo2/run/train.py.

General hardware requirements: See Hardware and Software Prerequisites — BioNeMo requires GPUs with Compute Capability ≥8.0 (Ampere or higher).

Fine-tuning tutorial: Check out the fine-tuning tutorial notebook for a complete walkthrough (currently covers the 1b model; the same train_evo2 command works for 7b with --model-size 7b).

Closing this out, but feel free to reopen if you have further questions!