RuntimeError: Error(s) in loading state_dict for InternViTModel/GPTVLModel

[Vintage-Echo]

Great job!

When I try to perform ViT weight conversion on the GPU-Megatron framework, I do not find the corresponding script.

> bash scripts/megatron/convert_model_intern_vit.sh

It’s seem to have some problems when I use other scripts. Could you provide the verified script for internViT-convert? Thanks!

> RuntimeError: Error(s) in loading state_dict for InternViTModel

[shenyunhang]

HI @Vintage-Echo
Please use this script to convert the vision model.

bash scripts/modellink/convert_model_intern_vit.sh

The vision model parts are the same in the Megatron and Modellink training framework.

[Vintage-Echo]

Yes, I have used this script before, but the problem still occurs. Is the environment configuration incorrect? When I used the docker shenyunhang/pytorch:24.11-py3_2024-1224，it's still occurd the similiar error:

RuntimeError:Error(s) in loading state_dict for GPTVLModel.

Where should I start to locate the problem? Looking forward to your reply, thanks a lot.

[shenyunhang]

I have verified this script and it works well.
And nothing should be related to GPTVLModel in the convert_model_intern_vit.sh.
It would be better to provide more logs to locate the problem.

[Vintage-Echo]

Yes, the problem might be caused by the loading of Qwen2.5-14B-Instruct_tp8pp1_te. The log file is as follows:

> [rank6]: Traceback (most recent call last):
> [rank6]:   File "/docker-longvita/data_local//Long-VITA//long_vita_megatron/pretrain_long_vita.py", line 1054, in <module>
> [rank6]:     pretrain(train_valid_test_datasets_provider,
> [rank6]:   File "/docker-longvita/data_local/Long-VITA/third_party/Megatron-LM_core_r0.7.0/megatron/training/training.py", line 227, in pretrain
> [rank6]:     model, optimizer, opt_param_scheduler = setup_model_and_optimizer(
> [rank6]:                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^
> [rank6]:   File "docker-longvita/data_local/Long-VITA/third_party/Megatron-LM_core_r0.7.0/megatron/training/training.py", line 517, in setup_model_and_optimizer
> [rank6]:     args.iteration, args.num_floating_point_operations_so_far = load_checkpoint(
> [rank6]:                                                                 ^^^^^^^^^^^^^^^^
> [rank6]:   File "docker-longvita/data_local/Long-VITA/third_party/Megatron-LM_core_r0.7.0/megatron/training/checkpointing.py", line 767, in load_checkpoint
> [rank6]:     model[0].load_state_dict(state_dict['model'], strict=strict)
> [rank6]:   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 2584, in load_state_dict
> [rank6]:     raise RuntimeError(
> [rank6]: RuntimeError: Error(s) in loading state_dict for GPTVLModel:
> [rank6]: 	Unexpected key(s) in state_dict: "decoder.layers.1.self_attention.linear_proj.weight", "decoder.layers.1.self_attention.linear_qkv.layer_norm_weight", "decoder.layers.1.self_attention.linear_qkv.weight", "decoder.layers.1.self_attention.linear_qkv.bias", "decoder.layers.1.mlp.linear_fc1.layer_norm_weight", "decoder.layers.1.mlp.linear_fc1.weight", "decoder.layers.1.mlp.linear_fc2.weight", "decoder.layers.2.self_attention.linear_proj.weight", "decoder.layers.2.self_attention.linear_qkv.layer_norm_weight", "decoder.layers.2.self_attention.linear_qkv.weight", "decoder.layers.2.self_attention.linear_qkv.bias", "decoder.layers.2.mlp.linear_fc1.layer_norm_weight", "decoder.layers.2.mlp.linear_fc1.weight", "decoder.layers.2.mlp.linear_fc2.weight", "decoder.layers.3.self_attention.linear_proj.weight", "decoder.layers.3.self_attention.linear_qkv.layer_norm_weight", "decoder.layers.3.self_attention.linear_qkv.weight", "decoder.layers.3.self_attention.linear_qkv.bias", "decoder.layers.3.mlp.linear_fc1.layer_norm_weight", "decoder.layers.3.mlp.linear_fc1.weight", "decoder.layers.3.mlp.linear_fc2.weight",

I temporarily bypass this problem by setting it to null, but I still do not know why the error is reported.

[shenyunhang]

It seems the model is not built with Transformer Engine, but the weights are converted for the Transformer Engine.
You can find the model structure in the log file and check for mismatches.

[Vintage-Echo]

I rechecked the model loading weight and restored the num-layers to 48，8 GPUs it was able to load and run smoothly.

However, I printed the corresponding parameter number in model_provider through torchinfo.summary, 8 GPUs params 15,100,018,688; 4 GPUs 15,090,577,408; 2 GPUs 15,085,856,768；1 GPU 15,083,496,448 (Number of parameters = Single card parameter x Number of cards).

The number of parameters is not exactly the same, but the difference is proportional. Are there any other parameters besides language model, vision model, and vision projector?