Pixtral 12B test_all_models_torch[mistral/pixtral/pytorch-single_device-inference] low PCC lately

[kmabeeTT]

This is a vision/language RED model with low PCC lately that our team owns and needs some debug.

Some more details

This test tests/runner/test_models.py::test_all_models_torch[mistral/pixtral/pytorch-single_device-inference] was passing prior to ~Dec 17 but then regressed to 0.96 PCC and then to 0.93 around Dec 25 where PCC was disabled via #2680

specific_test_case                                                                                                                          model_type  group       arch         bringup_status             model_status  pcc                     pcc_thres  pcc_en   parallelism      time      guidance
test_all_models_torch[mistral/pixtral/pytorch-single_device-inference]                                                                      vision      red         p150         INCORRECT_RESULT           PASSING       0.9310660697632953      0.99       PCC_DIS  single_device    160.218   N/A
test_all_models_torch[mistral/pixtral/pytorch-tensor_parallel-inference]                                                                    vision      red         n300-llmbox  INCORRECT_RESULT           PASSING       0.9760126530058753      0.99       PCC_DIS  tensor_parallel  189.437   N/A

It's failing for p150 (single_device) and n300-llmbox (tensor_parallel) but single_device will be easier to debug.

Assign to @devisettymahidhar608 can you please take a look? Some bisecting could make sense based on above datapoints, maybe.

[devisettymahidhar608]

Hi @kmabeeTT , @AleksKnezevic

I have few doubts regarding the model:

I have observed that the only variant present in loader.py is "mistral-community/pixtral-12b" and the loader.py is not formatted correctly.
Upon inspecting the load_inputs function, it returns the following keys: input_ids and attention_mask.

While running the model from Hugging Face in Google Colab, I observed a difference in the input keys being passed to the model.

The following keys are being sent to the model: ['input_ids', 'attention_mask', 'pixel_values', 'image_sizes'].

loader.py only sends input_ids and attention_mask; pixel_values and image_sizes are not included.

Questions:

1. Which approach should I follow: using the input keys from Hugging Face, or the ones defined in loader.py?

2. Can I raise a draft PR that includes formatting changes to loader.py, and then update the load_inputs function later based on your recommended input approach from question 1?

[kmabeeTT]

Hi @devisettymahidhar608 - good catch. Don't understand all the details, but I looked back in git history to see who added the model, and it seems when this model was added originally (in tt-torch project, before porting to tt-forge-models) we were using the chat_template originally but then during a transformers uplift last year to transformers==4.52 it started to output the 4 keys you mention, but this was an issue for some reason, so hardcoded 2 input keys was used instead of chat_template by change from @ddilbazTT. Talked about here: tenstorrent/tt-torch#904 (comment) - There is also a brief comment that got removed here saying:

    # The proper way of handling the inputs is to use the processor. However, starting transformers >= 4.50,
    # processor returns a different dict than what the model expects.
    # The following inputs was generated using the processor from transformers 4.47

Given this, I'm not sure... PCC debug is still valid since it was passing with high PCC and then regressed. But, It seems like using the chat_template is the right idea, so I'd say to give it a try and see if it helps, or if it causes more problems like Defne saw last year.

[devisettymahidhar608]

Sure @kmabeeTT ,
While applying processor.apply_chat_template(chat), I did not encounter the error mentioned above. The model runs successfully during input creation; however, runtime errors occur when executing the model.
I’ll debug the error and then raise a new PR. Does that sound fine?
Thanks.

[kmabeeTT]

Sounds good, thank you @devisettymahidhar608

[devisettymahidhar608]

Hi @kmabeeTT ,
I have raised a draft PR to reformat the code and corrected loader.py
While running the model on the P150 machine, I encountered some runtime errors and have raised an issue ticket.

Thanks