only chatglm4 report this question for kv_cache

[jjzhu0579]

Traceback (most recent call last):
File "/share/home/aim/aim_zhujj/qa_glm.py", line 121, in
response = query_bot(d['query'], context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/share/home/aim/aim_zhujj/qa_glm.py", line 90, in query_bot
generation_output = model.generate(
^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/transformers/generation/utils.py", line 2024, in generate
result = self._sample(
^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/transformers/generation/utils.py", line 2982, in _sample
outputs = self(**model_inputs, return_dict=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/share/home/aim/aim_zhujj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 894, in forward
transformer_outputs = self.transformer(
^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/share/home/aim/aim_zhujj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 788, in forward
hidden_states, presents, all_hidden_states, all_self_attentions = self.encoder(
^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/share/home/aim/aim_zhujj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 617, in forward
layer_ret = layer(
^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/share/home/aim/aim_zhujj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 519, in forward
attention_output, kv_cache = self.self_attention(
^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/share/home/aim/aim_zhujj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 386, in forward
key_layer = torch.cat((cache_k, key_layer), dim=2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected Tensor as element 0 in argument 0, but got str

[yixuantt]

Hi. This may be an error related to the inference process (transformers repo) of ChatGLM. Please check the implication of generation. 

[jjzhu0579]

Hi. This may be an error related to the inference process (transformers repo) of ChatGLM. Please check the implication of generation.

can you give me your py file which use the GPT-4

[yixuantt]

the script using GPT-4 just replace query_bot with query_api, which is the same as the following: 

try:
        response = openai.chat.completions.create(
            temperature=temperature,
            model="gpt-model-name",  
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": message}
            ]
        )
        print(response)
        return response.choices[0].message.content
    except Exception as e:
        print("OpenAI error: An error occurred: " + str(e))
        time.sleep(20)  # It's usually better to handle retries with a back-off strategy
        return query_openai(message)  # Recursive call on error

[jjzhu0579]

the script using GPT-4 just replace query_bot with query_api, which is the same as the following: 

try:
        response = openai.chat.completions.create(
            temperature=temperature,
            model="gpt-model-name",  
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": message}
            ]
        )
        print(response)
        return response.choices[0].message.content
    except Exception as e:
        print("OpenAI error: An error occurred: " + str(e))
        time.sleep(20)  # It's usually better to handle retries with a back-off strategy
        return query_openai(message)  # Recursive call on error

thankyou,I am a graduate student in the field of medical modeling and I am interested in researching Rag related topics. Can I add you on WeChat
or can you give me all the link to the models you use

[yixuantt]

Sorry, I can’t directly share the links to the models. I downloaded all of them from Hugging Face. Maybe you can send an email to the model provider asking for access. Good luck!