I can not seem to find certain sections in Nemo 2.0 for yaml config

[paniabhisek]

The complete data section of megatron_gpt_config.yaml is not present in Nemo 2.0 Data API.

Similarly how do I make sure all the configuration of megatron_gpt_config.yaml can be converted to Nemo 2.0. I did lot of things but some are still remaining.

[ashors1]

Hi, thanks for the question! Most of the settings controlled from the data section of the NeMo 1.0 configs map to parameters in the PreTrainingDataModule or FineTuningDataModule. Could you share the particular settings you're struggling to migrate?

[paniabhisek]

The following ones

  ##Offloading Activations/Weights to CPU
  cpu_offloading: False
  cpu_offloading_num_layers: ${sum:${.num_layers},-1} #This value should be between [1,num_layers-1] as we don't want to offload the final layer's activations and expose any offloading duration for the final layer
  cpu_offloading_activations: True
  cpu_offloading_weights: True

  reset_lr: False # Set to True to reset learning rate to initial learning rate. Only supported with distributed optmizer and megatron_amp_O2.
  reset_lr_steps: False # Set to True to adjust learning rate's max_steps and decay_steps by subtracting number of steps already completed at the checkpoint.

I will add more later.

[ashors1]

I see. All of the CPU offloading settings are part of MCore's TransformerConfig. NeMo's GPTConfig forwards all of the TransformerConfig settings to MCore, so you should be able to configure those from your GPTConfig directly.

For the LR-related settings, I think you're right that reset_lr is no longer supported. Could you explain your use case for enabling it?

[paniabhisek]

Thank you @ashors1 . Can you please tell me about the following? I do not have particular reason for any of the following. I was transferring to Nemo 2, so I was trying to do all of the config options.

mcore_gpt: True

rampup_batch_size: null # Should be a list of 3 values:
encoder_seq_length: 1024

megatron_amp_O2: True # Enable O2-level automatic mixed precision using main parameters
grad_allreduce_chunk_size_mb: 125

grad_div_ar_fusion: True # Fuse grad division into torch.distributed.all_reduce. Only used with O2 and no pipeline parallelism..
get_attention_mask_from_fusion: True # When using fused softmax it will create the attention mask so we won't copy it to the pipeline stages.

seed: 1234
apex_transformer_log_level: 30 # Python logging level displays logs with severity greater than or equal to this
gradient_as_bucket_view: True # PyTorch DDP argument. Allocate gradients in a contiguous bucket to save memory (less fragmentation and buffer memory)

data_impl: mmap
mmap_bin_files: True
skip_warmup: True
dataloader_type: single # cyclic
validation_drop_last: True # Set to false if the last partial validation samples is to be consumed
shuffle_documents: True # Set to False to disable documents shuffling. Sample index will still be shuffled

[paniabhisek]

I was looking at their respective classes to find the options. But it was not straightforward. I think we can put it in the docs where we can find those options