Adding prompts as files to promptfoo config #19
Conversation
Rolland-He
left a comment
Rolland-He
left a comment
There was a problem hiding this comment.
Hi @wkukka1, nice work! The code changes look good to me. As for the promptfoo updates, it’s a bit tricky to trace the diff. From what I see, they refactor config files from scenarios: to prompts:, which meets the goal. So good overall!
david-yz-liu
left a comment
david-yz-liu
left a comment
There was a problem hiding this comment.
@wkukka1 nice work; I left some inline comments. Please also update the remote tests (to have a list of file prompts).
README.md
Outdated
| | `--submission_type` | Type of submission (from `arg_options.FileType`) | ❌ | | ||
| | `--prompt` | Pre-defined prompt name or file path to custom prompt file | ❌ **| | ||
| | `--prompt_text` | Additional string text prompt that can be fed to model. | ❌ ** | | ||
| | `--prompt_text` | Additional string text prompt that can be fed to model or standalone prompt. | ❌ ** | |
There was a problem hiding this comment.
It's good to update the documentation here, but let's split this up. (1) here, just say "String prompt"; (2) expanding on the ** note below, say that if both --prompt and --prompt_text are provided, the prompt text argument is appended to the file contents (or something to that effect).
Similar comment for the help message as well.
ai_feedback/__main__.py
Outdated
|
|
||
| prompt_content = "" | ||
|
|
||
| prompt = {} |
There was a problem hiding this comment.
I don't think these changes are necessary. The only place the variable prompt is used is in the image scope, and that branch already defines prompt explicitly.
3 participants
Updates:
prompt_textto be able to act as a standalone promptprompts:instead ofscenariospromptfoo_test_runnerto useprompt_textarg