Skip to content

Commit 624d7f4

Browse files
wkukka1wkukka1
authored
Adding prompts as files to promptfoo config (#19)
1 parent fcb53ed commit 624d7f4

15 files changed

+460
-462
lines changed

‎README.md‎

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -26,24 +26,24 @@ For the image scope, the program takes up to two files, depending on the prompt
2626
- Saves response output in Markdown format with a predefined template or prints to stdout.
2727

2828
## Argument Details
29-
| Argument | Description | Required |
30-
|----------------------|-------------------------------------------------------------------|----------|
31-
| `--submission_type` | Type of submission (from `arg_options.FileType`) | ❌ |
32-
| `--prompt` | Pre-defined prompt name or file path to custom prompt file | ❌ **|
33-
| `--prompt_text` | Additional string text prompt that can be fed to model. | ❌ ** |
34-
| `--scope` | Processing scope (`image` or `code` or `text`) | ✅ |
35-
| `--submission` | Submission file path | ✅ |
36-
| `--question` | Specific question to evaluate | ❌ |
37-
| `--model` | Model type (from `arg_options.Models`) | ✅ |
38-
| `--output` | File path for where to record the output | ❌ |
39-
| `--solution` | File path for the solution file | ❌ |
40-
| `--test_output` | File path for the file containing the results from tests | ❌ |
41-
| `--submission_image` | File path for the submission image file | ❌ |
42-
| `--solution_image` | File path for the solution image file | ❌ |
29+
| Argument | Description | Required |
30+
|----------------------|---------------------------------------------------------------------|----------|
31+
| `--submission_type` | Type of submission (from `arg_options.FileType`) | ❌ |
32+
| `--prompt` | Pre-defined prompt name or file path to custom prompt file | ❌ **|
33+
| `--prompt_text` | String prompt | ❌ ** |
34+
| `--scope` | Processing scope (`image` or `code` or `text`) | ✅ |
35+
| `--submission` | Submission file path | ✅ |
36+
| `--question` | Specific question to evaluate | ❌ |
37+
| `--model` | Model type (from `arg_options.Models`) | ✅ |
38+
| `--output` | File path for where to record the output | ❌ |
39+
| `--solution` | File path for the solution file | ❌ |
40+
| `--test_output` | File path for the file containing the results from tests | ❌ |
41+
| `--submission_image` | File path for the submission image file | ❌ |
42+
| `--solution_image` | File path for the solution image file | ❌ |
4343
| `--system_prompt` | Pre-defined system prompt name or file path to custom system prompt | ❌ |
44-
| `--llama_mode` | How to invoke deepSeek-v3 (choices in `arg_options.LlamaMode`) | ❌ |
45-
| `--output_template` | Output template file (from `arg_options.OutputTemplate) | ❌ |
46-
** One of either `--prompt` or `--prompt_text` must be selected.
44+
| `--llama_mode` | How to invoke deepSeek-v3 (choices in `arg_options.LlamaMode`) | ❌ |
45+
| `--output_template` | Output template file (from `arg_options.OutputTemplate) | ❌ |
46+
** One of either `--prompt` or `--prompt_text` must be selected. If both are provided, `--prompt_text` will be appended to the contents of the file specified by `--prompt`.
4747

4848
## Scope
4949
The program supports three scopes: code or text or image. Depending on which is selected, the program supports different models and prompts tailored for each option.

‎ai_feedback/__main__.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -215,7 +215,6 @@ def main() -> int:
215215
args.submission_type = detect_submission_type(args.submission)
216216

217217
prompt_content = ""
218-
219218
system_instructions = load_system_prompt_content(args.system_prompt)
220219

221220
if args.prompt:

‎ai_feedback/helpers/constants.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
HELP_MESSAGES = {
33
"submission_type": "The format of the submission file (e.g., Jupyter notebook, Python script).",
44
"prompt": "Pre-defined prompt name (from ai_feedback/data/prompts/user/) or file path to custom prompt file.",
5-
"prompt_text": "Additional messages to concatenate to the prompt.",
5+
"prompt_text": "The string prompt that is sent to the model",
66
"scope": "The section of the assignment the model should analyze (e.g., code or image).",
77
"submission": "The file path for the submission file.",
88
"solution": "The file path for the solution file.",

‎promptfoo/promptfoo_test_runner.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,8 @@ def call_api(prompt: str, context: dict, metadata: dict) -> dict:
3434
options["scope"],
3535
"--model",
3636
options["model"],
37-
"--prompt",
38-
options['prompt'],
37+
"--prompt_text",
38+
prompt,
3939
"--llama_mode",
4040
"server",
4141
"--output_template",

‎promptfoo/tests/codellama_tests/codellama_code_tests.yaml‎

Lines changed: 56 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -6,59 +6,59 @@ defaultTest:
66
model: codellama:latest
77
scope: code
88

9-
scenarios:
10-
- config:
11-
- vars: { prompt: code_overall }
12-
tests:
13-
- vars:
14-
submission_file: test_submissions/csc263_opt_connected/correct_submission/correct_submission.py
15-
solution_file: test_submissions/csc263_opt_connected/solution.py
16-
17-
- vars:
18-
submission_file: test_submissions/csc263_opt_connected/fail_submission/fail_submission.py
19-
solution_file: test_submissions/csc263_opt_connected/solution.py
20-
21-
22-
- vars:
23-
submission_file: test_submissions/csc263_opt_connected/incorrect_algo_submission/incorrect_algo_submission.py
24-
solution_file: test_submissions/csc263_opt_connected/solution.py
25-
26-
- vars:
27-
submission_file: test_submissions/csc263_opt_connected/style_issues_submission/style_issues_submission.py
28-
solution_file: test_submissions/csc263_opt_connected/solution.py
29-
30-
- vars:
31-
submission_file: test_submissions/csc108/correct_submission/correct_submission.py
32-
solution_file: test_submissions/csc108/solution.py
33-
34-
- vars:
35-
submission_file: test_submissions/csc108/correctness_1_submission/correctness_1_submission.py
36-
solution_file: test_submissions/csc108/solution.py
37-
38-
- vars:
39-
submission_file: test_submissions/csc108/correctness_2_submission/correctness_2_submission.py
40-
solution_file: test_submissions/csc108/solution.py
41-
42-
- vars:
43-
submission_file: test_submissions/csc108/efficiency_submission/efficiency_submission.py
44-
solution_file: test_submissions/csc108/solution.py
45-
46-
- vars:
47-
submission_file: test_submissions/csc108/style_submission/style_submission.py
48-
solution_file: test_submissions/csc108/solution.py
49-
50-
- vars:
51-
submission_file: test_submissions/gac_example/correct_submission/correct_submission.py
52-
solution_file: test_submissions/gac_example/solution.py
53-
54-
- vars:
55-
submission_file: test_submissions/gac_example/fail_submission/fail_submission.py
56-
solution_file: test_submissions/gac_example/solution.py
57-
58-
- vars:
59-
submission_file: test_submissions/gac_example/inefficient_submission/inefficient_submission.py
60-
solution_file: test_submissions/gac_example/solution.py
61-
62-
- vars:
63-
submission_file: test_submissions/gac_example/partial_correct_submission/partial_correct_submission.py
64-
solution_file: test_submissions/gac_example/solution.py
9+
prompts:
10+
- file://../../../ai_feedback/data/prompts/user/code_overall.md
11+
12+
tests:
13+
- vars:
14+
submission_file: test_submissions/csc263_opt_connected/correct_submission/correct_submission.py
15+
solution_file: test_submissions/csc263_opt_connected/solution.py
16+
17+
- vars:
18+
submission_file: test_submissions/csc263_opt_connected/fail_submission/fail_submission.py
19+
solution_file: test_submissions/csc263_opt_connected/solution.py
20+
21+
22+
- vars:
23+
submission_file: test_submissions/csc263_opt_connected/incorrect_algo_submission/incorrect_algo_submission.py
24+
solution_file: test_submissions/csc263_opt_connected/solution.py
25+
26+
- vars:
27+
submission_file: test_submissions/csc263_opt_connected/style_issues_submission/style_issues_submission.py
28+
solution_file: test_submissions/csc263_opt_connected/solution.py
29+
30+
- vars:
31+
submission_file: test_submissions/csc108/correct_submission/correct_submission.py
32+
solution_file: test_submissions/csc108/solution.py
33+
34+
- vars:
35+
submission_file: test_submissions/csc108/correctness_1_submission/correctness_1_submission.py
36+
solution_file: test_submissions/csc108/solution.py
37+
38+
- vars:
39+
submission_file: test_submissions/csc108/correctness_2_submission/correctness_2_submission.py
40+
solution_file: test_submissions/csc108/solution.py
41+
42+
- vars:
43+
submission_file: test_submissions/csc108/efficiency_submission/efficiency_submission.py
44+
solution_file: test_submissions/csc108/solution.py
45+
46+
- vars:
47+
submission_file: test_submissions/csc108/style_submission/style_submission.py
48+
solution_file: test_submissions/csc108/solution.py
49+
50+
- vars:
51+
submission_file: test_submissions/gac_example/correct_submission/correct_submission.py
52+
solution_file: test_submissions/gac_example/solution.py
53+
54+
- vars:
55+
submission_file: test_submissions/gac_example/fail_submission/fail_submission.py
56+
solution_file: test_submissions/gac_example/solution.py
57+
58+
- vars:
59+
submission_file: test_submissions/gac_example/inefficient_submission/inefficient_submission.py
60+
solution_file: test_submissions/gac_example/solution.py
61+
62+
- vars:
63+
submission_file: test_submissions/gac_example/partial_correct_submission/partial_correct_submission.py
64+
solution_file: test_submissions/gac_example/solution.py

‎promptfoo/tests/deepseek_r1_tests/deepseek_r1_code_tests.yaml‎

Lines changed: 55 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -6,58 +6,58 @@ defaultTest:
66
model: deepSeek-R1:70B
77
scope: code
88

9-
scenarios:
10-
- config:
11-
- vars: { prompt: code_feedback_r1 }
12-
tests:
13-
- vars:
14-
submission_file: test_submissions/csc263_opt_connected/correct_submission/correct_submission.py
15-
solution_file: test_submissions/csc263_opt_connected/solution.py
16-
17-
- vars:
18-
submission_file: test_submissions/csc263_opt_connected/fail_submission/fail_submission.py
19-
solution_file: test_submissions/csc263_opt_connected/solution.py
20-
21-
- vars:
22-
submission_file: test_submissions/csc263_opt_connected/incorrect_algo_submission/incorrect_algo_submission.py
23-
solution_file: test_submissions/csc263_opt_connected/solution.py
24-
25-
- vars:
26-
submission_file: test_submissions/csc263_opt_connected/style_issues_submission/style_issues_submission.py
27-
solution_file: test_submissions/csc263_opt_connected/solution.py
28-
29-
- vars:
30-
submission_file: test_submissions/csc108/correct_submission/correct_submission.py
31-
solution_file: test_submissions/csc108/solution.py
32-
33-
- vars:
34-
submission_file: test_submissions/csc108/correctness_1_submission/correctness_1_submission.py
35-
solution_file: test_submissions/csc108/solution.py
36-
37-
- vars:
38-
submission_file: test_submissions/csc108/correctness_2_submission/correctness_2_submission.py
39-
solution_file: test_submissions/csc108/solution.py
40-
41-
- vars:
42-
submission_file: test_submissions/csc108/efficiency_submission/efficiency_submission.py
43-
solution_file: test_submissions/csc108/solution.py
44-
45-
- vars:
46-
submission_file: test_submissions/csc108/style_submission/style_submission.py
47-
solution_file: test_submissions/csc108/solution.py
48-
49-
- vars:
50-
submission_file: test_submissions/gac_example/correct_submission/correct_submission.py
51-
solution_file: test_submissions/gac_example/solution.py
52-
53-
- vars:
54-
submission_file: test_submissions/gac_example/fail_submission/fail_submission.py
55-
solution_file: test_submissions/gac_example/solution.py
56-
57-
- vars:
58-
submission_file: test_submissions/gac_example/inefficient_submission/inefficient_submission.py
59-
solution_file: test_submissions/gac_example/solution.py
60-
61-
- vars:
62-
submission_file: test_submissions/gac_example/partial_correct_submission/partial_correct_submission.py
63-
solution_file: test_submissions/gac_example/solution.py
9+
prompts:
10+
- file://../../../ai_feedback/data/prompts/user/code_feedback_r1.md
11+
12+
tests:
13+
- vars:
14+
submission_file: test_submissions/csc263_opt_connected/correct_submission/correct_submission.py
15+
solution_file: test_submissions/csc263_opt_connected/solution.py
16+
17+
- vars:
18+
submission_file: test_submissions/csc263_opt_connected/fail_submission/fail_submission.py
19+
solution_file: test_submissions/csc263_opt_connected/solution.py
20+
21+
- vars:
22+
submission_file: test_submissions/csc263_opt_connected/incorrect_algo_submission/incorrect_algo_submission.py
23+
solution_file: test_submissions/csc263_opt_connected/solution.py
24+
25+
- vars:
26+
submission_file: test_submissions/csc263_opt_connected/style_issues_submission/style_issues_submission.py
27+
solution_file: test_submissions/csc263_opt_connected/solution.py
28+
29+
- vars:
30+
submission_file: test_submissions/csc108/correct_submission/correct_submission.py
31+
solution_file: test_submissions/csc108/solution.py
32+
33+
- vars:
34+
submission_file: test_submissions/csc108/correctness_1_submission/correctness_1_submission.py
35+
solution_file: test_submissions/csc108/solution.py
36+
37+
- vars:
38+
submission_file: test_submissions/csc108/correctness_2_submission/correctness_2_submission.py
39+
solution_file: test_submissions/csc108/solution.py
40+
41+
- vars:
42+
submission_file: test_submissions/csc108/efficiency_submission/efficiency_submission.py
43+
solution_file: test_submissions/csc108/solution.py
44+
45+
- vars:
46+
submission_file: test_submissions/csc108/style_submission/style_submission.py
47+
solution_file: test_submissions/csc108/solution.py
48+
49+
- vars:
50+
submission_file: test_submissions/gac_example/correct_submission/correct_submission.py
51+
solution_file: test_submissions/gac_example/solution.py
52+
53+
- vars:
54+
submission_file: test_submissions/gac_example/fail_submission/fail_submission.py
55+
solution_file: test_submissions/gac_example/solution.py
56+
57+
- vars:
58+
submission_file: test_submissions/gac_example/inefficient_submission/inefficient_submission.py
59+
solution_file: test_submissions/gac_example/solution.py
60+
61+
- vars:
62+
submission_file: test_submissions/gac_example/partial_correct_submission/partial_correct_submission.py
63+
solution_file: test_submissions/gac_example/solution.py

‎promptfoo/tests/deepseek_r1_tests/deepseek_r1_text_tests.yaml‎

Lines changed: 31 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -6,34 +6,34 @@ defaultTest:
66
model: deepSeek-R1:70B
77
scope: text
88

9-
scenarios:
10-
- config:
11-
- vars: { prompt: text_analyze_r1 }
12-
tests:
13-
- vars:
14-
submission_file: test_submissions/data_collection_ethics_module/average_submission/average_submission.txt
15-
solution_file: test_submissions/data_collection_ethics_module/solution.txt
16-
17-
- vars:
18-
submission_file: test_submissions/data_collection_ethics_module/excellent_submission/excellent_submission.txt
19-
solution_file: test_submissions/data_collection_ethics_module/solution.txt
20-
21-
- vars:
22-
submission_file: test_submissions/data_collection_ethics_module/off_topic_submission/off_topic_submission.txt
23-
solution_file: test_submissions/data_collection_ethics_module/solution.txt
24-
25-
- vars:
26-
submission_file: test_submissions/data_collection_ethics_module/weak_submission/weak_submission.txt
27-
solution_file: test_submissions/data_collection_ethics_module/solution.txt
28-
29-
- vars:
30-
submission_file: test_submissions/csc373_eft_optimality_proof/fail_submission/fail_submission.pdf
31-
solution_file: test_submissions/csc373_eft_optimality_proof/solution.pdf
32-
33-
- vars:
34-
submission_file: test_submissions/csc373_eft_optimality_proof/incomplete_submission/incomplete_submission.pdf
35-
solution_file: test_submissions/csc373_eft_optimality_proof/solution.pdf
36-
37-
- vars:
38-
submission_file: test_submissions/csc373_eft_optimality_proof/induction_submission/induction_submission.pdf
39-
solution_file: test_submissions/csc373_eft_optimality_proof/solution.pdf
9+
prompts:
10+
- file://../../../ai_feedback/data/prompts/user/text_analyze_r1.md
11+
12+
tests:
13+
- vars:
14+
submission_file: test_submissions/data_collection_ethics_module/average_submission/average_submission.txt
15+
solution_file: test_submissions/data_collection_ethics_module/solution.txt
16+
17+
- vars:
18+
submission_file: test_submissions/data_collection_ethics_module/excellent_submission/excellent_submission.txt
19+
solution_file: test_submissions/data_collection_ethics_module/solution.txt
20+
21+
- vars:
22+
submission_file: test_submissions/data_collection_ethics_module/off_topic_submission/off_topic_submission.txt
23+
solution_file: test_submissions/data_collection_ethics_module/solution.txt
24+
25+
- vars:
26+
submission_file: test_submissions/data_collection_ethics_module/weak_submission/weak_submission.txt
27+
solution_file: test_submissions/data_collection_ethics_module/solution.txt
28+
29+
- vars:
30+
submission_file: test_submissions/csc373_eft_optimality_proof/fail_submission/fail_submission.pdf
31+
solution_file: test_submissions/csc373_eft_optimality_proof/solution.pdf
32+
33+
- vars:
34+
submission_file: test_submissions/csc373_eft_optimality_proof/incomplete_submission/incomplete_submission.pdf
35+
solution_file: test_submissions/csc373_eft_optimality_proof/solution.pdf
36+
37+
- vars:
38+
submission_file: test_submissions/csc373_eft_optimality_proof/induction_submission/induction_submission.pdf
39+
solution_file: test_submissions/csc373_eft_optimality_proof/solution.pdf

0 commit comments

Comments
 (0)