Skip to content

Conversation

@ArquintL
Copy link
Member

@ArquintL ArquintL commented Mar 13, 2025

This PR splits up the one big CI job we have into 7 (example) and executes 4 of them in parallel to speed up our CI runs (~16 vs. 22 min).
The main change is that this PR splits sbt test into executing GobraTests for each subdirectory in regression and executing all remaining test classes. I.e., we currently have the following jobs:

  • execute GobraTests on directory regressions/examples
  • execute GobraTests on directory regressions/features
  • execute GobraTests on directory regressions/issues
  • execute all remaining test classes
    To achieve this behavior, I've introduced a tag GobraTestsTag, which unfortunately has to be Java code. This tag allows us to run only test classes either with or without this tag.
    Furthermore, the CI automatically detects the subfolders in regressions and spawns jobs accordingly such that we do not miss test cases if someone adds further subdirectories.

I've marked this PR as a draft as we can discuss the following points / decisions:

  • We currently used sbt test and use now sbt testOnly to execute the test cases (instead of using a precompiled JAR file) which involves compiling the test classes in each of the 4 jobs mentioned above, which adds some overhead. Do we want to stick to this? I'm currently testing the alternative on this branch
  • The execution time varies quite a bit between these 4 jobs (~6.5, 8.25, 3.5, and 4.75min). Should we split differently? If so how? Or should we further partition regression/features?
  • Should we try to further parallelize executing the tests within each job by instructing sbt to do so?

@ArquintL ArquintL marked this pull request as draft March 13, 2025 08:28
@ArquintL ArquintL requested a review from jcp19 March 13, 2025 08:28
@ArquintL ArquintL added the enhancement New feature or request label Mar 13, 2025
@jcp19
Copy link
Contributor

jcp19 commented Mar 28, 2025

before we move on with merging this, it would be cool to understand what problems we get when enabling parallel tests in sbt and see if there's a way to fix them. That might be the simplest solution

@ArquintL
Copy link
Member Author

ArquintL commented Apr 4, 2025

@jcp19 I've tried using the ParallelTestExecution trait that executes tests within a test suite in parallel on branch parallelize-GobraTests. While I can see an impressive speedup for just a few testcases, the execution time of the entire test suite significantly increases for some reason. Furthermore, I get a lot of NoClassDefFoundError errors at runtime when executing the tests via sbt.
I made several changes to even be able to run the GobraTests in that branch and, thus, I'd suggest to merge them while keeping their execution sequential

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants