Replies: 1 comment 2 replies
-
A good rule of thumb with Futureverse, but also in other parallel frameworks, is, if possible. to make as little assumptions on where things will be running. Another one is to not make assumptions about being able to return to the same parallel worker again. For example, for some parallel backends, the workers are "transient" where they are spun up to process a single task and then shut down immediately after. I expect the latter will become more and more common as we go forward. So, with this in mind, code that first "initiates" workers and then makes a second round of calls for processing tasks will not work on the workers that are transient. Any hands-on example of this would be if you used the future.callr backend as in; future::plan(future.callr::callr, workers = n_cores)
# Initialize GSW package in each worker
foreach::foreach (i = 1:n_cores, .options.future = list(seed = TRUE)) %dofuture% {
gswdesign::gswdesign_setup()
}
assignments = foreach::foreach (i = 1:m, .options.future = list(seed = TRUE)) %dofuture% {
gswdesign::sample_gs_walk(
X = X, phi = 0.1, balanced = TRUE,
treatment_probs = 0.5)
} Whatever settings you do to the parallel workers in the first Instead, you want do something like: assignments = foreach::foreach (i = 1:m, .options.future = list(seed = TRUE)) %dofuture% {
## Initiate the GWS Julia engine
gswdesign::gswdesign_setup()
gswdesign::sample_gs_walk(
X = X, phi = 0.1, balanced = TRUE,
treatment_probs = 0.5)
} That code would run distributed anywhere in the world without you having to make assumption on the workers, as long as the workers have R, future, gswdesign, and Julia installed. Now, it looks like gswdesign::gswdesign_setup()
gswdesign::gswdesign_setup() in the same R session, it will do the same setup twice. Ideally, assignments = foreach::foreach (i = 1:m, .options.future = list(seed = TRUE)) %dofuture% {
## Initiate the GWS Julia engine, if not already done
if (!gswdesign::gswdesign_already_setup()) gswdesign::gswdesign_setup()
gswdesign::sample_gs_walk(
X = X, phi = 0.1, balanced = TRUE,
treatment_probs = 0.5)
} There's an internal |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I am using R package "gswdesign", and I would to call it parallel using "future". In order to call the function in the R package, there is a setup script which sets up a Julia environment. I would like to know how to pass this to the parallel sessions in an elegant way, and make it work. Any help is appreciated.
Sample code:
Packages and setup script
Generate data
Generate single design (calling gswdesign once)
Generate m designs using future (using 1 core)
Generate m designs using future (2 cores)
FAILS: Error in .gsw_intenv$julia$eval("true") : attempt to apply non-function
Generate multiple designs using future (2 cores)
WORKS - Initialize GSW package in each worker first
Is there a more elegant/fast way to pass this setup script/environment to the sessions?
A link to what "gswdesign::gswdesign_setup()" does:
https://github.com/fsavje/gswdesign-R/blob/master/R/gswdesign-package.R
Beta Was this translation helpful? Give feedback.
All reactions