[OpenVINO][Draft]support Hunyuan LLM #1429
Conversation
| return dummy_inputs | ||
|
|
||
|
|
||
| class HunyuanDummyPastKeyValuesGenerator(DummyPastKeyValuesGenerator): |
There was a problem hiding this comment.
why not use instead MistralDummyPastKeyValuesGenerator and set instead normalized_config.head_dim
| self.random_float_tensor(shape, framework=framework, dtype=float_dtype), | ||
| ) | ||
| for _ in range(self.num_layers) | ||
| ] |
There was a problem hiding this comment.
would you mind adding a test as well
There was a problem hiding this comment.
yes, I will add it once the release version of transformers support this model.
There was a problem hiding this comment.
please add tests for inference: https://github.com/huggingface/optimum-intel/blob/main/tests/openvino/test_modeling.py
Co-authored-by: Roman Kazantsev <[email protected]>
I will add it after this PR |
Let us anticipate that PR to be merged. Then you can add tests to this PR. No need to have several PRs and separate implementation and tests. We need to make sure that inference works. Best regards, |
tests/openvino/test_decoder.py
Outdated
| <<<<<<< HEAD | ||
| "ernie4_5": 2, | ||
| "hunyuan_v1_dense": 2, | ||
| ======= | ||
| >>>>>>> upstream/main |
There was a problem hiding this comment.
artifacts of merge, please fix
|
|
||
| @register_in_tasks_manager("hunyuan_v1_dense", *["text-generation", "text-generation-with-past"], library_name="transformers") | ||
| class HunyuanOpenVINOConfig(TextDecoderWithPositionIdsOnnxConfig): | ||
| MIN_TRANSFORMERS_VERSION = "4.55.0.dev0" |
There was a problem hiding this comment.
not sure that we need dev0 suffix
There was a problem hiding this comment.
since optimum-intel does not support Transformers 4.56, this PR can only work with the commit for Transformers 4.55.
git+https://github.com/huggingface/transformers@4970b23cedaf745f963779b4eae68da281e8c6ca
There was a problem hiding this comment.
tests for modelling to test generate() method is needed as well
There was a problem hiding this comment.
tests for modelling to test
generate()method is needed as well
its already covered in test_compare_to_transformers i think
3 participants
No description provided.