You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+11-5Lines changed: 11 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -367,7 +367,6 @@ This python package can be used as a dependency in the Markus Autotester, in ord
367
367
### Markus Test Scripts
368
368
- /markus_test_scripts contains scripts which can be uploaded to the autotester in order to generate LLM Feedback
369
369
- Currently, only openAI and Claude models are supported.
370
-
- The /test_submissions directory has mock assignment submissions, solutions, and test files, used for testing the markus integration. These files can be submitted on the markus autotester along with the llm script files.
371
370
- Within these llm script files, the models and prompts used can be changed by editing the command line arguments, through the run_llm() function.
372
371
373
372
Files:
@@ -378,8 +377,8 @@ Files:
378
377
379
378
#### Python AutoTester Usage
380
379
##### Code Scope
381
-
1. Ensure the student has submitted a submission file (_submission suffixed).
382
-
2. Ensure the instructor has submitted a solution file (_solution suffixed), llm_helpers.py (located in /markus_test_scripts), and python_tester_llm_code.py (located in /markus_test_scripts). Instructor can also upload another pytest file which can be run as its own test group.
380
+
1. Ensure the student has submitted a submission file.
381
+
2. Ensure the instructor has submitted a solution file, llm_helpers.py (located in /markus_test_scripts), and python_tester_llm_code.py (located in /markus_test_scripts). Instructor can also upload another pytest file which can be run as its own test group.
383
382
3. Ensure the submission import statement in python_tester_llm_code.py matches the name of the student's submission file name.
384
383
4. Create a Python Autotester Test Group to run the LLM File.
385
384
5. In the Package Requirements section of the Test Group Settings for the LLM file, put:
@@ -394,6 +393,13 @@ Along with any other packages that the submission or solution file uses.
394
393
##### Text Scope
395
394
- Do the same as the code scope, but ensure that the student submission and instructor solution are .pdf files with the same naming assumption. Also, ensure that python_tester_llm_pdf.py is uploaded as the test script.
396
395
396
+
##### AI Tester Usage
397
+
1. In the Autotest settings of the assignment, click Add Tester and select the `ai` option.
398
+
2. Fill in all required arguments for the AI tester.
399
+
3. Upload any related files (e.g., JSON schema files, custom prompts, or configuration files).
400
+
4. Ensure the MarkUs Autotester Docker container has the API keys defined in an .env file and that these variables are specified in the docker-compose.yml file.
401
+
5. Ensure the Timeout is set to 120 seconds or longer.
402
+
397
403
#### Running Python Autotester Examples
398
404
##### CNN Example
399
405
- Look at the /test_submissions/cnn_example directory for the following files
@@ -417,8 +423,8 @@ Along with any other packages that the submission or solution file uses.
417
423
- Student uploads: student_pdf_submission.pdf
418
424
419
425
#### Custom Tester Usage
420
-
1. Ensure the student has submitted a submission file (_submission suffixed).
421
-
2. Ensure the instructor has submitted a solution file (_solution suffixed) and custom_tester_llm_code.sh (located in /markus_test_scripts). Instructor can also upload another script used to run its own test group. (See below for GGR274 Example.)
426
+
1. Ensure the student has submitted a submission file.
427
+
2. Ensure the instructor has submitted a solution file and custom_tester_llm_code.sh (located in /markus_test_scripts). Instructor can also upload another script used to run its own test group. (See below for GGR274 Example.)
0 commit comments