Skip to content

Commit d5c6dcd

Browse files
authored
Updated MarkUs integration steps in README (#33)
1 parent 90a7f36 commit d5c6dcd

File tree

1 file changed

+11
-5
lines changed

1 file changed

+11
-5
lines changed

README.md

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -367,7 +367,6 @@ This python package can be used as a dependency in the Markus Autotester, in ord
367367
### Markus Test Scripts
368368
- /markus_test_scripts contains scripts which can be uploaded to the autotester in order to generate LLM Feedback
369369
- Currently, only openAI and Claude models are supported.
370-
- The /test_submissions directory has mock assignment submissions, solutions, and test files, used for testing the markus integration. These files can be submitted on the markus autotester along with the llm script files.
371370
- Within these llm script files, the models and prompts used can be changed by editing the command line arguments, through the run_llm() function.
372371

373372
Files:
@@ -378,8 +377,8 @@ Files:
378377

379378
#### Python AutoTester Usage
380379
##### Code Scope
381-
1. Ensure the student has submitted a submission file (_submission suffixed).
382-
2. Ensure the instructor has submitted a solution file (_solution suffixed), llm_helpers.py (located in /markus_test_scripts), and python_tester_llm_code.py (located in /markus_test_scripts). Instructor can also upload another pytest file which can be run as its own test group.
380+
1. Ensure the student has submitted a submission file.
381+
2. Ensure the instructor has submitted a solution file, llm_helpers.py (located in /markus_test_scripts), and python_tester_llm_code.py (located in /markus_test_scripts). Instructor can also upload another pytest file which can be run as its own test group.
383382
3. Ensure the submission import statement in python_tester_llm_code.py matches the name of the student's submission file name.
384383
4. Create a Python Autotester Test Group to run the LLM File.
385384
5. In the Package Requirements section of the Test Group Settings for the LLM file, put:
@@ -394,6 +393,13 @@ Along with any other packages that the submission or solution file uses.
394393
##### Text Scope
395394
- Do the same as the code scope, but ensure that the student submission and instructor solution are .pdf files with the same naming assumption. Also, ensure that python_tester_llm_pdf.py is uploaded as the test script.
396395

396+
##### AI Tester Usage
397+
1. In the Autotest settings of the assignment, click Add Tester and select the `ai` option.
398+
2. Fill in all required arguments for the AI tester.
399+
3. Upload any related files (e.g., JSON schema files, custom prompts, or configuration files).
400+
4. Ensure the MarkUs Autotester Docker container has the API keys defined in an .env file and that these variables are specified in the docker-compose.yml file.
401+
5. Ensure the Timeout is set to 120 seconds or longer.
402+
397403
#### Running Python Autotester Examples
398404
##### CNN Example
399405
- Look at the /test_submissions/cnn_example directory for the following files
@@ -417,8 +423,8 @@ Along with any other packages that the submission or solution file uses.
417423
- Student uploads: student_pdf_submission.pdf
418424

419425
#### Custom Tester Usage
420-
1. Ensure the student has submitted a submission file (_submission suffixed).
421-
2. Ensure the instructor has submitted a solution file (_solution suffixed) and custom_tester_llm_code.sh (located in /markus_test_scripts). Instructor can also upload another script used to run its own test group. (See below for GGR274 Example.)
426+
1. Ensure the student has submitted a submission file.
427+
2. Ensure the instructor has submitted a solution file and custom_tester_llm_code.sh (located in /markus_test_scripts). Instructor can also upload another script used to run its own test group. (See below for GGR274 Example.)
422428
3. In the Markus Autotesting terminal:
423429
``` bash
424430
docker exec -it -u 0 markus-autotesting-server-1 /bin/bash

0 commit comments

Comments
 (0)