System prompt added twice with "user" instead of "system" label?

[jens321]

Very cool work!

When adding the system prompt here: https://github.com/balrog-ai/BALROG/blob/main/balrog/prompt_builder/history.py#L85-L89, is it intended to be added twice (i.e. when icl_episodes is False)? Also, should the label be "system" instead of "user"?

Thanks!

[DavidePaglieri]

Hi, thank you for the kind words, and thanks for spotting this!

The double system prompt is a bug accidentally introduced in the few-shot learning PR #4, fixed it in #21.

The label "user" is correct here, even though this functions as a system prompt, we still pass it as if it was sent from the user, this is because not all LLMs support the use of a "system" prompt.