Fix: Add metadata to bf16 safetensors for compatibility with transformers #749
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue:
When using HuggingFace Transformers versions earlier than v4.47.1, loading the DeepSeek-V3 bf16 weights generated by
fp8_cast_bf16.py
raises the following error:How to reproduce:
This issue occurs because the transformers library expects a specific metadata entry
{"format": "pt"}
in each safetensors file.Proposed solution:
Similar to transformers https://github.com/huggingface/transformers/blob/6966fa190172b48b2fb46fe4552a13b943e692cf/src/transformers/modeling_utils.py#L3132 I changed line 88 of
fp8_cast_bf16.py
to:Tested locally, and the model successfully loads after applying the fix. Example uploaded weights: https://huggingface.co/tflsxyy/DeepSeek-V3-bf16
Even though not many people will load bf16 weight of DeepSeek-V3/R1, and this error is fixed by transformers library after v4.48.0 https://github.com/huggingface/transformers/blob/6bc0fbcfa7acb6ac4937e7456a76c2f7975fefec/src/transformers/modeling_utils.py#L506 but I think there are still some people testing with the bf16 weights and using transformers library before v4.47.1, this code needs to change.