Integrate SysMoBench benchmark via Git Subtree #10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… 119a763 git-subtree-dir: benchmarks/sysmobench/sysmobench_core git-subtree-split: 119a763435c4b7266a27cc8f6741239d0f4e4347
…s/sysmobench/sysmobench_core'
- Add gpt-4o model configuration to models.yaml - Fix setup_tools.py to use shutil.move instead of os.rename This resolves 'Invalid cross-device link' error when /tmp is on different filesystem
Qian-Cheng-nju
changed the title
xuafeng
requested review from
tareknaser and
tianyin
and removed request for
tareknaser
xuafeng
requested changes
Nov 14, 2025
Collaborator
xuafeng
left a comment
xuafeng
left a comment
There was a problem hiding this comment.
Thanks a lot for your great contribution to this project. The integration is great. I left a few suggestions.
| ### Contribute to existing Benchmarks | ||
| The easiest way to contribute is to add more tasks to existing benchmarks. For example, you can add more questions to the course exam benchmark or more projects to the course project benchmark. You can add more system algorithm design problems into algorithm design benchmark. Please follow the existing format and structure for adding new tasks. You can also improve the existing benchmarks by adding more advanced evaluators with improved metrics. | ||
|
|
||
| ### Creating New Benchmarks |
Collaborator
There was a problem hiding this comment.
As discussed in the slack, can you please help add a section called "### Porting your benchmark", and using SysMoBench as example? Thanks a lot.
Collaborator
Author
There was a problem hiding this comment.
I’ve written a draft of the porting benchmark guide. When you have time, I would greatly appreciate your review. Please feel free to let me know if anything should be improved. Thank you very much! See commit 72b6916
Collaborator
|
Resolves #7 |
Couen
pushed a commit
to Couen/system-intelligence-benchmark
that referenced
this pull request
Jan 22, 2026
…ench Integrate SysMoBench benchmark via Git Subtree
tareknaser
pushed a commit
that referenced
this pull request
Feb 5, 2026
Integrate SysMoBench benchmark via Git Subtree
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add SysMoBench benchmark - evaluates AI models' ability to generate correct TLA+ formal specifications for real-world concurrent and distributed systems.