fix(courseexam): parse_exam_markdown to increment start_instance_id

Signed-off-by: Tarek <[email protected]>

Author: tareknaser

benchmarks/courseexam_bench/data/example_course_2024_midterm/exam.md

@@ -23,10 +23,15 @@
 What state is a process in when it is waiting for I/O to complete?
 
 A. Running
+
 B. Ready
+
 C. Blocked
+
 D. Terminated
 
+Your answer should be a single letter only (A, B, C, or D).
+
 ```json
 {
   "problem_id": "1",
@@ -44,6 +49,8 @@ D. Terminated
 
 True or False: A race condition occurs when multiple threads access shared data concurrently and at least one thread modifies the data.
 
+Your answer should be either "True" or "False" only. No extra text.
+
 ```json
 {
   "problem_id": "2",
@@ -61,6 +68,8 @@ True or False: A race condition occurs when multiple threads access shared data
 
 Explain the purpose of a Translation Lookaside Buffer (TLB) in a virtual memory system.
 
+Your answer should be a brief explanation (about 2-3 sentences).
+
 ```json
 {
   "problem_id": "3",
@@ -78,6 +87,8 @@ Explain the purpose of a Translation Lookaside Buffer (TLB) in a virtual memory
 
 Describe the two phases of the two-phase commit protocol.
 
+Your answer should include a brief description of each phase (about 1-2 sentences each).
+
 ```json
 {
   "problem_id": "4",
@@ -97,17 +108,21 @@ Describe the two phases of the two-phase commit protocol.
 Which of the following operations modify the inode? (Select all that apply)
 
 A. Changing file permissions
+
 B. Reading file contents
+
 C. Changing file size
 
+Your answer should be a comma-separated list of letters only (no extra text). For example: "A,B"
+
 ```json
 {
   "problem_id": "5",
   "points": 5,
   "type": "ExactMatch",
   "tags": ["operating-systems", "file-systems"],
-  "answer": "A,C",
-  "comments": "Multiple correct answers but NO partial credit. Only exact match \"A,C\" gets 5 points."
+  "answer": "A, C",
+  "comments": "Multiple correct answers but NO partial credit. Only exact match \"A, C\" gets 5 points."
 }
 ```
 
@@ -118,17 +133,21 @@ C. Changing file size
 Which statements about CPU scheduling are true? (Select all that apply)
 
 A. Round-robin can lead to starvation
+
 B. SJF minimizes average waiting time
+
 C. Priority scheduling can have priority inversion
 
+Your answer should be a comma-separated list of letters only (no extra text). For example: "A, B"
+
 ```json
 {
   "problem_id": "6",
   "points": 10,
   "type": "Freeform",
   "tags": ["operating-systems", "scheduling"],
-  "answer": "B,C",
-  "llm_judge_instructions": "Correct: B,C. Award 10 points for B,C. Award 6 points for only B or only C. Award 0 if A is selected (incorrect).",
+  "answer": "B, C",
+  "llm_judge_instructions": "Correct: B, C. Award 10 points for B, C. Award 6 points for only B or only C. Award 0 if A is selected (incorrect).",
   "comments": "Multiple choice with partial credit. Freeform type with rubric allows rewarding incomplete but correct answers while penalizing wrong choices."
 }
 ```
@@ -160,6 +179,8 @@ Refer to the Raft Algorithm Reference.
 
 True or False: A candidate must receive votes from a majority of servers to become leader.
 
+Your answer should be either "True" or "False" only. No extra text.
+
 ```json
 {
   "problem_id": "8",
@@ -180,6 +201,8 @@ Refer to the Raft Algorithm Reference.
 
 What are the three possible outcomes when a candidate runs for election? List all three.
 
+Your answer should list the three outcomes in a single response (one to two sentences each).
+
 ```json
 {
   "problem_id": "9",
@@ -199,6 +222,8 @@ What are the three possible outcomes when a candidate runs for election? List al
 
 True or False: In Paxos, a proposer must receive responses from a majority of acceptors to achieve consensus.
 
+Your answer should be either "True" or "False" only. No extra text.
+
 ```json
 {
   "problem_id": "10.1",
@@ -216,6 +241,8 @@ True or False: In Paxos, a proposer must receive responses from a majority of ac
 
 True or False: The CAP theorem states that a distributed system can simultaneously guarantee Consistency, Availability, and Partition tolerance.
 
+Your answer should be either "True" or "False" only. No extra text.
+
 ```json
 {
   "problem_id": "10.2",

benchmarks/courseexam_bench/prepare_dataset.py

@@ -3,7 +3,7 @@
 from pathlib import Path
 
 
-def parse_exam_markdown(exam_md_path):
+def parse_exam_markdown(exam_md_path, start_instance_id):
     content = exam_md_path.read_text()
 
     exam_metadata_match = re.search(r"```json\n(\{[^`]+?\})\n```", content)
@@ -12,7 +12,7 @@ def parse_exam_markdown(exam_md_path):
     sections = content.split("\n---\n")[1:]
 
     questions = []
-    instance_id = 1
+    instance_id = start_instance_id
 
     for section in sections:
         json_blocks = re.findall(r"```json\n(\{[^`]+?\})\n```", section)
@@ -67,7 +67,7 @@ def parse_exam_markdown(exam_md_path):
             questions.append(question)
             instance_id += 1
 
-    return exam_metadata, questions
+    return exam_metadata, questions, instance_id
 
 
 def main():
@@ -76,6 +76,7 @@ def main():
 
     all_exams_metadata = []
     all_questions = []
+    next_instance_id = 1
 
     for exam_folder in data_dir.iterdir():
         if not exam_folder.is_dir() or exam_folder.name in [
@@ -88,7 +89,9 @@ def main():
         if not exam_md.exists():
             continue
 
-        exam_metadata, questions = parse_exam_markdown(exam_md)
+        exam_metadata, questions, next_instance_id = parse_exam_markdown(
+            exam_md, next_instance_id
+        )
         all_exams_metadata.append(exam_metadata)
         all_questions.extend(questions)