[OpenVINO]support Hunyuan LLM #1429

openvino-dev-samples · 2025-08-20T13:15:29Z

What does this PR do?

Conversion cmd-line for tencent/Hunyuan-7B-Instruct or Hunyuan-7B-Instruct:

optimum-cli export openvino --model tencent/Hunyuan-7B-Instruct Hunyuan-7B-Instruct-ov --weight-format fp16 --task text-generation-with-past

Inference of Hunyuan-7B-Instruct using OpenVINO backend:

from optimum.intel.openvino import OVModelForCausalLM
from transformers import AutoTokenizer
import os
import re

model_name_or_path = "tencent/Hunyuan-7B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
model = OVModelForCausalLM.from_pretrained(model_name_or_path, device_map="auto")  # You may want to use bfloat16 and/or move to GPU here
messages = [
    {"role": "user", "content": "Write a short summary of the benefits of regular exercise"},
]
tokenized_chat = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True,return_tensors="pt",
                                                enable_thinking=True # Toggle thinking mode (default: True)
                                                )
                                                
outputs = model.generate(tokenized_chat.to(model.device), max_new_tokens=2048)

output_text = tokenizer.decode(outputs[0])
print("output_text=",output_text)
think_pattern = r'<think>(.*?)</think>'
think_matches = re.findall(think_pattern, output_text, re.DOTALL)

answer_pattern = r'<answer>(.*?)</answer>'
answer_matches = re.findall(answer_pattern, output_text, re.DOTALL)

think_content = [match.strip() for match in think_matches][0]
answer_content = [match.strip() for match in answer_matches][0]
print(f"thinking_content:{think_content}\n\n")
print(f"answer_content:{answer_content}\n\n")

Before submitting

Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

update

echarlaix · 2025-08-22T10:51:49Z

optimum/exporters/openvino/model_configs.py

        return dummy_inputs
+
+
+class HunyuanDummyPastKeyValuesGenerator(DummyPastKeyValuesGenerator):


why not use instead MistralDummyPastKeyValuesGenerator and set instead normalized_config.head_dim

echarlaix · 2025-08-22T10:52:02Z

optimum/exporters/openvino/model_configs.py

+                self.random_float_tensor(shape, framework=framework, dtype=float_dtype),
+            )
+            for _ in range(self.num_layers)
+        ]


would you mind adding a test as well

yes, I will add it once the release version of transformers support this model.

optimum/exporters/openvino/model_configs.py

rkazants

please add tests for inference: https://github.com/huggingface/optimum-intel/blob/main/tests/openvino/test_modeling.py

Co-authored-by: Roman Kazantsev <[email protected]>

openvino-dev-samples · 2025-09-15T14:28:21Z

please add tests for inference: https://github.com/huggingface/optimum-intel/blob/main/tests/openvino/test_modeling.py

I will add it after this PR

rkazants · 2025-09-16T05:02:26Z

I will add it after this PR

Let us anticipate that PR to be merged. Then you can add tests to this PR. No need to have several PRs and separate implementation and tests. We need to make sure that inference works.

Best regards,
Roman

rkazants · 2025-10-13T03:15:01Z

tests/openvino/test_decoder.py

+<<<<<<< HEAD
+        "ernie4_5": 2,
+        "hunyuan_v1_dense": 2,
+=======
+>>>>>>> upstream/main


artifacts of merge, please fix

rkazants · 2025-10-13T03:15:37Z

optimum/exporters/openvino/model_configs.py

+
+@register_in_tasks_manager("hunyuan_v1_dense", *["text-generation", "text-generation-with-past"], library_name="transformers")
+class HunyuanOpenVINOConfig(TextDecoderWithPositionIdsOnnxConfig):
+    MIN_TRANSFORMERS_VERSION = "4.55.0.dev0"


not sure that we need dev0 suffix

since optimum-intel does not support Transformers 4.56, this PR can only work with the commit for Transformers 4.55.

git+https://github.com/huggingface/transformers@4970b23cedaf745f963779b4eae68da281e8c6ca

rkazants · 2025-10-16T07:26:57Z

tests/openvino/test_decoder.py

tests for modelling to test generate() method is needed as well

tests for modelling to test generate() method is needed as well

its already covered in test_compare_to_transformers i think

HuggingFaceDocBuilderDev · 2025-12-02T14:33:55Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

echarlaix · 2025-12-02T15:15:00Z

Hi @openvino-dev-samples #1529 adds support for transformers v4.56 so we can soon merge this PR. looks like some test is currently failing would you mind taking a look ?

echarlaix

Thanks a lot @openvino-dev-samples ! Waiting for #1541 to be merged before we can merge this PR, hopefully this can be done soon cc @rkazants

optimum/exporters/openvino/model_configs.py

Co-authored-by: Ella Charlaix <[email protected]>

rkazants · 2025-12-15T19:21:02Z

tests/openvino/test_decoder.py

+    if is_transformers_version("<", "4.56.0"):
+        SUPPORTED_ARCHITECTURES += ("qwen", "chatglm", "chatglm4")
+


why it is needed. It looks to be fixed during transition to latest transformers

its just copied from legacy version, and let me remove it.

openvino-dev-samples added 5 commits July 28, 2025 18:34

update hunyuan patcher

800aae9

update hunyuan patcher

73193b2

update hunyuan patcher

150db50

align with tranformers

f051f08

update

2c01397

update

echarlaix reviewed Aug 22, 2025

View reviewed changes

openvino-dev-samples added 3 commits September 10, 2025 08:44

update with MistralDummyPastKeyValuesGenerator

a7afdae

update test case

116b098

update test case

c4f7cc3

rkazants reviewed Sep 15, 2025

View reviewed changes

optimum/exporters/openvino/model_configs.py Outdated Show resolved Hide resolved

rkazants suggested changes Sep 15, 2025

View reviewed changes

Update optimum/exporters/openvino/model_configs.py

4127d04

Co-authored-by: Roman Kazantsev <[email protected]>

openvino-dev-samples added 2 commits September 27, 2025 08:26

rebase

159574d

Update model_configs.py

c50b7a9

rkazants reviewed Oct 13, 2025

View reviewed changes

remove artifacts of merge

4a34ab5

rkazants reviewed Oct 16, 2025

View reviewed changes

echarlaix added 4 commits December 2, 2025 15:23

merge main

7095543

add missing

df81c99

fix style

cc870f7

set min transformers version to 4.56

1a54cd8

openvino-dev-samples added 2 commits December 2, 2025 18:10

update test case of hunyuan

553a58b

Merge branch 'main' into hunyuan

d25153b

openvino-dev-samples changed the title ~~[OpenVINO][Draft]support Hunyuan LLM~~ [OpenVINO]support Hunyuan LLM Dec 3, 2025

Merge branch 'main' into hunyuan

2f49bff

echarlaix approved these changes Dec 11, 2025

View reviewed changes

optimum/exporters/openvino/model_configs.py Outdated Show resolved Hide resolved

optimum/exporters/openvino/model_configs.py Outdated Show resolved Hide resolved

openvino-dev-samples and others added 2 commits December 11, 2025 22:59

Update optimum/exporters/openvino/model_configs.py

2a9bd5d

Co-authored-by: Ella Charlaix <[email protected]>

Update optimum/exporters/openvino/model_configs.py

6d879da

Co-authored-by: Ella Charlaix <[email protected]>

rkazants reviewed Dec 15, 2025

View reviewed changes

rkazants requested a review from popovaan December 15, 2025 19:21

remove the legacy patch

75750ac

		return dummy_inputs


		class HunyuanDummyPastKeyValuesGenerator(DummyPastKeyValuesGenerator):

		if is_transformers_version("<", "4.56.0"):
		SUPPORTED_ARCHITECTURES += ("qwen", "chatglm", "chatglm4")

[OpenVINO]support Hunyuan LLM #1429

Are you sure you want to change the base?

[OpenVINO]support Hunyuan LLM #1429

Conversation

openvino-dev-samples commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rkazants left a comment

Choose a reason for hiding this comment

Uh oh!

openvino-dev-samples commented Sep 15, 2025

Uh oh!

rkazants commented Sep 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Dec 2, 2025

Uh oh!

echarlaix commented Dec 2, 2025

Uh oh!

echarlaix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

openvino-dev-samples commented Aug 20, 2025 •

edited

Loading