Skip to content

regression tests & benchmark#929

Closed
epolack wants to merge 1 commit into
JuliaMolSim:masterfrom
epolack:pkgbenchmark
Closed

regression tests & benchmark#929
epolack wants to merge 1 commit into
JuliaMolSim:masterfrom
epolack:pkgbenchmark

Conversation

@epolack

@epolack epolack commented Dec 12, 2023 •
edited
Loading

Copy link
Copy Markdown
Collaborator
  • If the action is to merge on master, maybe compare to previous known release? Otherwise, makes mostly sense only for PR.
  • Locally, the results are consistent. On the CI the target is way to fast, maybe because for the action to make sense, I have to run against the exact same version. I tested with retune, and the results are more logical; so I am keeping it that way for now:
    no-retune v.s. retune
  • Separate load testing from regression?
  • For the humongous tests it still need some work for it to be flexible and easily usable; e.g., allow user to chose baseline, allow to use retune option.
  • Silence tests (Using Suppressor to silence tests #957)
  • Add regression tests for Julia Nightly?
  • Document for the developer.
  • For humongous tests, retune not useful
  • Have @benchmarkable directly in code?
  • Density and ad tests are too inconsistent between run. Increase load.
  • use DFTK-testproblems as submodule? Modify CI with 
    with:
  submodules: true
  • The github-bot does not comment on the PR, even if indicated as so. May be a permission issue, but do we really need this feature? Also, will be messed up if two concurrent jobs.

Also: added @testset to continue even if some tests fails…

@epolack epolack linked an issue Dec 12, 2023 that may be closed by this pull request
Use BenchmarkCI to automatically detect major runtime regressions #923
Open
@epolack epolack force-pushed the pkgbenchmark branch 7 times, most recently from 61daf9d to 59f1cc8 Compare December 12, 2023 11:24
epolack
epolack commented Dec 12, 2023
View reviewed changes
Comment thread .github/workflows/regression.yaml Outdated
- name: Run benchmarks
# Remove baseline once merged. Regression tests will only work after this is merged
# in master.
run: julia -e 'using BenchmarkCI; BenchmarkCI.judge(; baseline="HEAD", retune=true)'

@epolack epolack Dec 12, 2023

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not forget.

Comment thread benchmark/run.jl Outdated
Comment on lines +11 to +12
# Remove target once merged. Regression tests will only work after this is merged in master.
BenchmarkCI.judge(; baseline="HEAD~1")

@epolack epolack Dec 12, 2023

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not forget.

@epolack epolack force-pushed the pkgbenchmark branch 21 times, most recently from 48e9277 to 960b0b4 Compare December 12, 2023 13:57
@epolack epolack force-pushed the pkgbenchmark branch 4 times, most recently from 1f29d84 to 62f97a5 Compare December 19, 2023 17:17
@epolack epolack force-pushed the pkgbenchmark branch 19 times, most recently from 812b783 to 4eebd3f Compare December 22, 2023 17:46
@epolack epolack force-pushed the pkgbenchmark branch 2 times, most recently from d22cd80 to d50e4af Compare February 26, 2024 09:05
@mfherbst

mfherbst commented May 14, 2025

Copy link
Copy Markdown
Member

Superseeded by #1093.

@mfherbst mfherbst closed this May 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Use BenchmarkCI to automatically detect major runtime regressions

2 participants

@epolack @mfherbst