Skip to content

Add benchmarking framework and dataset scripts#1598

Open
mandel wants to merge 38 commits into
mainfrom
benchmarks
Open

Add benchmarking framework and dataset scripts#1598
mandel wants to merge 38 commits into
mainfrom
benchmarks

Conversation

@mandel

@mandel mandel commented May 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a benchmarking framework with benchmark runners and result analysis tooling
  • add dataset download scripts and benchmark implementations for FEVER, GSM8K, LiveCodeBench, Math500, and MBPP
  • document the benchmark workflow and datasets

Testing

  • not run

mandel and others added 29 commits May 27, 2026 13:57
@mandel
Add benchmarking framework and dataset scripts
f3e1cd2 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
pre-commit fixes in benchmarks
e623a0a 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Update dependencies
84df8b0 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Fix pre-commit
f015b7b 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Add comments
652dbad 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Fix pre-commit
9259e73 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Fix pre-commit with Python 3.11 and 3.12
6069a3e 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Add PPDL programs
5f5287c 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Download datasets in the datasets directory
87484e2 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Update get_dataset scripts
3d939aa 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Experiement config files
75aeefd 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Make the benchmarl less sensitive to the path
b504e25 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Update README
d472385 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Fix bench_analyzer
1f62cc8 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Benchmarks dependencies
2db16a7 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Update gitignore
3537deb 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
skip testing of benchmark programs
9ba190a 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@dependabot @mandel
chore(deps): bump openssl in /pdl-live-react/src-tauri (#1597)
b029919 
Bumps [openssl](https://github.com/rust-openssl/rust-openssl) from 0.10.78 to 0.10.79.
- [Release notes](https://github.com/rust-openssl/rust-openssl/releases)
- [Commits](rust-openssl/rust-openssl@openssl-v0.10.78...openssl-v0.10.79)

---
updated-dependencies:
- dependency-name: openssl
  dependency-version: 0.10.79
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update rust crate tauri to v2.11.1 [security] (#1600)
9ca3a07 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update dependency yaml to v2.9.0 (#1590)
52e5510 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update rust crate tokio to v1.52.3 (#1595)
36f7bf9 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update dependency react-router to v7.15.1 (#1596)
e265f26 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update dependency @tauri-apps/plugin-fs to v2.5.1 (#1588)
dbb1a7a 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update rust crate tauri-plugin-opener to v2.5.4 (#1592)
b70dbf0 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update dependency @tauri-apps/plugin-opener to v2.5.4 (#…
5732eb5 
…1589)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@dependabot @mandel
chore(deps-dev): update pymilvus requirement from ~=2.5 to >=2.5,<4.0 (…
9f3315e 
…#1601)

Updates the requirements on [pymilvus](https://github.com/milvus-io/pymilvus) to permit the latest version.
- [Release notes](https://github.com/milvus-io/pymilvus/releases)
- [Commits](milvus-io/pymilvus@v2.5.0...v3.0.0)

---
updated-dependencies:
- dependency-name: pymilvus
  dependency-version: 3.0.0
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update react monorepo to v19.2.15 (#1605)
dcbd311 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update dependency vite to v7.3.3 (#1604)
594ee75 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update dependency @types/node to v24.12.4 (#1602)
066333e 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
mandel and others added 9 commits May 27, 2026 13:57
@mandel
Add seaborn dependency to pyproject.toml (#1608)
c131258 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@mandel
Update README
c257771 
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@dependabot @mandel
chore(deps): bump openssl in /pdl-live-react/src-tauri (#1610)
acbf35f 
Bumps [openssl](https://github.com/rust-openssl/rust-openssl) from 0.10.79 to 0.10.80.
- [Release notes](https://github.com/rust-openssl/rust-openssl/releases)
- [Commits](rust-openssl/rust-openssl@openssl-v0.10.79...openssl-v0.10.80)

---
updated-dependencies:
- dependency-name: openssl
  dependency-version: 0.10.80
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update dependency @playwright/test to v1.60.0 (#1607)
847c0ae 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
fix(deps): update rust crate async-openai to 0.40.0 (#1609)
bbd48a3 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update rust crate minijinja to v2.20.0 (#1611)
158497d 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update dependency typescript-eslint to v8.59.4 (#1603)
ffa0741 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update rust crate serde_json to v1.0.150 (#1614)
4673f4e 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
@renovate @mandel
chore(deps): update tauri monorepo (#1606)
ba2da22 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Louis Mandel <lmandel@us.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mandel