Skip to content

Conversation

@qimcis
Copy link
Collaborator

@qimcis qimcis commented Jan 18, 2026

Description

Add the CS350 Fall 2018 Midterm exam dataset and associated solution file to the course exam benchmark.

Changes

  • Add exam directory at /home/qi/system-intelligence-benchmark/benchmarks/courseexam_bench/data/raw/cs350_fall_2018_midterm
  • Include solutions file F18-midterm-sol.pdf in the exam directory
  • Add exam metadata with ID cs350_fall_2018_midterm and note that there are no reference files

Testing

Prepared the dataset and verified processing by running: python3 courseexam/prepare.py

Checklist

  • Tests pass locally
  • Code follows project style guidelines
  • Documentation updated (if needed)

xuafeng and others added 30 commits November 5, 2025 18:10
…stinguish-api-keys

Distinguish the models used in the executor and evaluator
- Add gpt-4o model configuration to models.yaml
- Fix setup_tools.py to use shutil.move instead of os.rename
  This resolves 'Invalid cross-device link' error when /tmp is on different filesystem
tareknaser and others added 23 commits January 8, 2026 10:01
* modified the regular expression match

* added cs107_fall_2023_final

* updated one wrong tag

* Added pdf of the paper

* solved all the comments
* Add UIUC CS423 Fall 2025 Exams

* Fix json parsing

* Fix backtick

* Fix backtick 2

* Update benchmarks/courseexam_bench/data/cs_423_operating_systems_design_fall_2025_midterm/MP1.md

Co-authored-by: Copilot <[email protected]>

* Update benchmarks/courseexam_bench/data/cs_423_operating_systems_design_fall_2025_final/exam.md

Co-authored-by: Copilot <[email protected]>

---------

Co-authored-by: Xuan Feng <[email protected]>
Co-authored-by: Copilot <[email protected]>
…ce#54)

* add ostep processes-shell lab

* fix

* Update benchmarks/courselab_bench/data/cs537-projects-spring-2019/processes_shell/task.md

Co-authored-by: Copilot <[email protected]>

---------

Co-authored-by: Tarek Elsayed <[email protected]>
Co-authored-by: Copilot <[email protected]>
* added cmu15-213 data lab

* docs(courselab): add note about infrastructure restrictions

Signed-off-by: Tarek <[email protected]>

---------

Signed-off-by: Tarek <[email protected]>
Co-authored-by: Tarek <[email protected]>
* add cs537 fall 2021 final exam

* add institution

* fix

* add solutions

* update metadata

* add choice array

* avoid extra restrictions on LLM output

Signed-off-by: Tarek <[email protected]>

---------

Signed-off-by: Tarek <[email protected]>
Co-authored-by: Tarek <[email protected]>
Copy link
Collaborator

@777lefty 777lefty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some missing issues and format issues


## Question 2 [3 point(s)]

Explain why the following implementation of semaphore P is incorrect. Provide an example interaction between two threads that illustrates the problem.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code snippet for this question seems to be totally missed

n /= d;
printf("%d\n", d);
return 0;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better use table and code block to wrap them


## Question 7 [2 point(s)]

What concurrency problem does this program suffer from?
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code snippet provided in the background for all the subproblems of the problem is missed

```

---

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question 3 is skipped, but it seems to be feasible because it doesn't contain figure, and we can ask the agents to output table to represent stack


---

## Question 10 [2 point(s)]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Background information for all subproblems in problem 4 is missed

copy trapframe to stack
modify trapframe
enter usermode
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code block better?

```

---

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question 5 is skipped, seems to be feasible as well


---

## Question 15 [3 point(s)]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Background information for Question 6 is missed

@qimcis qimcis force-pushed the cs350-fall-2018-midterm branch from 67457ba to 0fc8bab Compare January 26, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants