Customized position_ids not working

[huchanwei123]

System Info

Hello,

I am trying to feed a customized position IDs to Llama model.
If I fed a customized position_ids vector, for example, [[0, 0, 1, 2, 2, 2]] means batch size = 1, 1st and 2nd tokens share the same position, and 3rd-5th tokens share the same position 2, this will cause an error.

The error seems to be located in the function prepare_inputs_for_generation in src/transformers/models/llama/modeling_llama.py, where the position_ids does not change as the cache_position increase, so the shape inconsistency occurs.

Is there any way to successfully feed a customized position ids to the model?
Thanks!

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. Give a customized position_ids as one of the input in model.generate()

Expected behavior

Size mismatch

[ArthurZucker]

Cc @gante

[codeslayed]

You can directly modify how position_ids are computed within your code before passing them to the model. Ensure that your custom position_ids are aligned with the expected shape and values.

[huchanwei123]

You can directly modify how position_ids are computed within your code before passing them to the model. Ensure that your custom position_ids are aligned with the expected shape and values.

Thanks for the reply.
Yes, I am aware that I can pass a customized position_ids to the model, and I believe the shape and values are correct.

Since the model generates token-by-token, feeding a customized position_ids causes an size mismatch error after the first token is generated.
After a little bit digging, I found that in the function prepare_inputs_for_generation, there is no implementation handling when position_ids is not None. As a result, during the generation of the second token, the shape of position_ids is still the same, but it is expected to match the shape of attention_mask.

I am not sure if I missed or misunderstood anything.
Thanks!

[zucchini-nlp]

It has a tracker here (#29149) and we had a PR a few months ago. Unfortunately the PR was too big and needed to be decomposed into parts, after which it went low in priority :(

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.