Problem with judge prompt for MM-MT-Bench

[gabegma]

Hi!

I'm trying to reproduce the MM-MT-Bench results, and I'm struggling a little bit. I believe that the judge's prompt in MM-MT-Bench is incorrect. Below is an example and my interrogations

1 - The user content is not being populated, as the code is looping on keys instead of values (hence we get rolecontent)
2 - The reference answer has the full dictionary instead of only the text - is that on purpose? The Pixtral paper doesn't mention that.
3 - The text often contains headers (#, ##, ###) which the model might confuse with ### User and ### Reference answer.
4 - Lastly, in the Pixtral paper, it is hinted that the image might be sent to the judge model, but this is not happening in the code either - is the paper or the code the source of truth?

Thank you so much for the help!

'<|The Start of Conversation with User|>

### User:
rolecontent

### Reference answer:
[{\'type\': \'text\', \'text\': "# Military Battalion Comparison (1990 - 2020)\\n\\nThis comparison chart outlines the variations in battalion numbers within different military branches – Armoured, Infantry, and Artillery – for a selection of countries between the years 1990 and 2020.\\n\\n## Key Trends and Statistics\\n\\n### Overall Reduction in Forces\\n- All highlighted countries have undergone substantial decreases in their total number of battalions across all branches, indicating a widespread trend of downsizing military forces over the span of three decades.\\n\\n### Germany\'s Dramatic Decrease\\n- Germany\'s reduction is particularly notable. In 1990, West Germany counted a total of 215 battalions:\\n  - **Armoured:** 85\\n  - **Infantry:** 67\\n  - **Artillery:** 63\\n- By 2020, unified Germany had only 33 battalions:\\n  - **Armoured:** 11\\n  - **Infantry:** 18\\n  - **Artillery:** 4\\n- This represents an approximate 85% cutback.\\n\\n### Shift in Force Composition\\n- Although all types saw reductions, shifts in the composition of forces occurred:\\n  - **Armoured** battalions faced the sharpest declines. Germany\'s armoured battalions dropped by 87%, from 85 to 11.\\n  - **Infantry** battalions generally experienced smaller reductions. Britain\'s infantry units reduced by 41%, from 58 to 34.\\n  - **Artillery** units were also significantly cut, often more so than infantry but to a lesser extent than armoured units. France\'s artillery battalions fell from 23 to 7, a 70% decrease.\\n\\n### U.S. EUCOM (European Command) Reductions\\n- The U.S. military footprint in Europe diminished substantially, with a reduction from 99 battalions in 1990 to merely 16 in 2020, equating to an 84% decline.\\n\\n### Relative Positions Unchanged\\n- Despite the overall reductions, the relative standings in terms of battalion counts have remained approximately constant. Germany and Italy retained more battalions than France and Britain both in 1990 and 2020.\\n\\n## Analysis\\n\\nThe revealed trends likely mirror shifts in military strategies, budget allocations, and the changing geopolitical climate after the Cold War\'s conclusion. The trend away from massed armoured formations infers a strategic pivot toward more nimble and deployable forces, potentially as a reaction to evolving threat landscapes and military operations in the post-Cold War environment."}]

### Assistant's answer:
This assistant's answer [placeholder]

<|The End of Conversation with User|>