LLMs palm reported an error regarding attention_mask

[jjzhu0579]

Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:25<00:25, 25.72s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:32<00:00, 14.84s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:32<00:00, 16.47s/it]
/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: clean_up_tokenization_spaces was not set. It will be set to True by default. This behavior will be depracted in transformers v4.45, and will be then set to False by default. For more details check this issue: huggingface/transformers#31884
warnings.warn(

0%| | 0/1194 [00:00<?, ?it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (2947 > 1024). Running this sequence through the model will result in indexing errors
/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/transformers/generation/configuration_utils.py:567: UserWarning: do_sample is set to False. However, temperature is set to 0.1 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset temperature. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
warnings.warn(
/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/transformers/generation/configuration_utils.py:567: UserWarning: do_sample is set to False. However, temperature is set to 0.1 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset temperature.
warnings.warn(
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's attention_mask to obtain reliable results.
Setting pad_token_id to eos_token_id:50256 for open-end generation.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's attention_mask to obtain reliable results.

0%| | 0/1194 [00:07<?, ?it/s]
Traceback (most recent call last):
File "/share/home/aim/aim_zhujj/qa_palm.py", line 117, in
response = query_bot(d['query'], context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/share/home/aim/aim_zhujj/qa_palm.py", line 86, in query_bot
generation_output = model.generate(
^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/transformers/generation/utils.py", line 2024, in generate
result = self._sample(
^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/transformers/generation/utils.py", line 2982, in _sample
outputs = self(**model_inputs, return_dict=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1315, in forward
transformer_outputs = self.transformer(
^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1037, in forward
attention_mask = _prepare_4d_causal_attention_mask_for_sdpa(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/transformers/modeling_attn_mask_utils.py", line 372, in _prepare_4d_causal_attention_mask_for_sdpa
ignore_causal_mask = AttentionMaskConverter._ignore_causal_mask_sdpa(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/aim_nuist/aim_zhujj/.conda/envs/new_env_name/lib/python3.12/site-packages/transformers/modeling_attn_mask_utils.py", line 279, in _ignore_causal_mask_sdpa
elif (is_training or not is_tracing) and torch.all(attention_mask == 1):
^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[jjzhu0579]

And after I solved this bug, the output of the model was also very poor

[jjzhu0579]