Skip to content

feat: add mirai parallelization #1314

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open

feat: add mirai parallelization #1314

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
c6f410a
...
be-marc Dec 3, 2024
e79da73
...
be-marc Mar 27, 2025
a9fa258
...
be-marc Apr 8, 2025
e39c244
...
be-marc Apr 17, 2025
b98ad9b
Merge branch 'main' into mirai
be-marc May 21, 2025
9ae87a4
...
be-marc May 22, 2025
67e6031
...
be-marc May 22, 2025
c050efa
...
be-marc May 22, 2025
1cdcfc7
...
be-marc May 22, 2025
597fa9b
...
be-marc May 22, 2025
63709a6
...
be-marc May 22, 2025
6c45a15
...
be-marc May 22, 2025
3d90df8
...
be-marc May 22, 2025
8ef3cbb
...
be-marc May 22, 2025
f511627
...
be-marc May 22, 2025
1198a7d
...
be-marc May 22, 2025
8437803
...
be-marc May 22, 2025
e28bc38
...
be-marc May 22, 2025
2ee1b76
...
be-marc May 22, 2025
2e60779
...
be-marc May 22, 2025
5f08eaa
...
be-marc May 22, 2025
db0af10
...
be-marc May 23, 2025
867d128
chore: remotes
be-marc May 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .Rbuildignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,5 @@
^cran-comments\.md$
^CRAN-SUBMISSION$
^benchmark$
^attic$
^.cursor$
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,3 +183,4 @@ revdep/
# misc
Meta/
Rplots.pdf
.cursor/
3 changes: 3 additions & 0 deletions DESCRIPTION
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,15 @@ Suggests:
codetools,
datasets,
future.callr,
mirai,
mlr3data,
progressr,
remotes,
RhpcBLASctl,
rpart,
testthat (>= 3.2.0)
Remotes:
mlr-org/mlr3misc@mirai
Encoding: UTF-8
Config/testthat/edition: 3
Config/testthat/parallel: false
Expand Down
2 changes: 1 addition & 1 deletion R/Learner.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -554,7 +554,7 @@ Learner = R6Class("Learner",
#'
#' @return `self` (invisibly).
encapsulate = function(method, fallback = NULL) {
assert_choice(method, c("none", "try", "evaluate", "callr"))
assert_choice(method, c("none", "try", "evaluate", "callr", "mirai"))

if (method != "none") {
assert_learner(fallback, task_type = self$task_type)
Expand Down
3 changes: 3 additions & 0 deletions R/helper_exec.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ future_map = function(n, FUN, ..., MoreArgs = list()) {
if (getOption("mlr3.debug", FALSE)) {
lg$info("Running experiments sequentially in debug mode with %i iterations", n)
mapply(FUN, ..., MoreArgs = MoreArgs, SIMPLIFY = FALSE, USE.NAMES = FALSE)
} else if (requireNamespace("mirai", quietly = TRUE) && mirai::status()$connections) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so this will automatically prefer mirai over future? Maybe there should be an option for this

Copy link
Member Author

@be-marc be-marc May 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only if the user also started a daemon with mirai::daemons()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only works with the default compute profile

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I didn't want to make it too complicated. What would be the best way to pass the compute profile? We also have to pass it to mirai_map() .

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest we expose a .compute argument to benchmark() and resample().

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not so happy with that because we are introducing parameters for a special backend. Moreover, this argument would then have to be added to all tuning and feature selection functions.

lg$debug("Running resample() via mirai with %i iterations", n)
mirai::collect_mirai(mirai::mirai_map(data.table(...), workhorse, .args = c(MoreArgs, list(is_sequential = FALSE))))
} else {
is_sequential = inherits(plan(), "sequential")
scheduling = if (!is_sequential && isTRUE(getOption("mlr3.exec_random", TRUE))) structure(TRUE, ordering = "random") else TRUE
Expand Down
Loading