How to pass setup script of an R package (gswdesign) to future

[nremenyi]

I am using R package "gswdesign", and I would to call it parallel using "future". In order to call the function in the R package, there is a setup script which sets up a Julia environment. I would like to know how to pass this to the parallel sessions in an elegant way, and make it work. Any help is appreciated.

Sample code:

Packages and setup script

library(JuliaCall)
library(future)
library(foreach)
library(doFuture)
library(gswdesign)
gswdesign::gswdesign_setup()

Generate data

set.seed(1)
n = 200
p = 5
X = matrix(sample(x = 1:100, size = n*p, replace = TRUE), ncol = p)

Generate single design (calling gswdesign once)

assignment = gswdesign::sample_gs_walk(
  X = X, phi = 0.1, balanced = TRUE,
  treatment_probs = 0.5)

Generate m designs using future (using 1 core)

m = 100
n_cores = 1
future::plan(future::multisession, workers = n_cores)

assignments = foreach::foreach (i = 1:m, .options.future = list(seed = TRUE)) %dofuture% { 
  gswdesign::sample_gs_walk(
    X = X, phi = 0.1, balanced = TRUE,
    treatment_probs = 0.5)
}

Generate m designs using future (2 cores)
FAILS: Error in .gsw_intenv$julia$eval("true") : attempt to apply non-function

n_cores = 2
future::plan(future::multisession, workers = n_cores)

assignments = foreach::foreach (i = 1:m, .options.future = list(seed = TRUE)) %dofuture% { 
  gswdesign::sample_gs_walk(
    X = X, phi = 0.1, balanced = TRUE,
    treatment_probs = 0.5)
}

Generate multiple designs using future (2 cores)
WORKS - Initialize GSW package in each worker first

n_cores = 2
future::plan(future::multisession, workers = n_cores)

# Initialize GSW package in each worker
foreach::foreach (i = 1:n_cores, .options.future = list(seed = TRUE)) %dofuture% {
  gswdesign::gswdesign_setup()
}

assignments = foreach::foreach (i = 1:m, .options.future = list(seed = TRUE)) %dofuture% { 
  gswdesign::sample_gs_walk(
    X = X, phi = 0.1, balanced = TRUE,
    treatment_probs = 0.5)
}

Is there a more elegant/fast way to pass this setup script/environment to the sessions?

A link to what "gswdesign::gswdesign_setup()" does:
https://github.com/fsavje/gswdesign-R/blob/master/R/gswdesign-package.R

[HenrikBengtsson]

A good rule of thumb with Futureverse, but also in other parallel frameworks, is, if possible. to make as little assumptions on where things will be running. Another one is to not make assumptions about being able to return to the same parallel worker again. For example, for some parallel backends, the workers are "transient" where they are spun up to process a single task and then shut down immediately after. I expect the latter will become more and more common as we go forward. So, with this in mind, code that first "initiates" workers and then makes a second round of calls for processing tasks will not work on the workers that are transient. Any hands-on example of this would be if you used the future.callr backend as in;

future::plan(future.callr::callr, workers = n_cores)

# Initialize GSW package in each worker
foreach::foreach (i = 1:n_cores, .options.future = list(seed = TRUE)) %dofuture% {
  gswdesign::gswdesign_setup()
}

assignments = foreach::foreach (i = 1:m, .options.future = list(seed = TRUE)) %dofuture% { 
  gswdesign::sample_gs_walk(
    X = X, phi = 0.1, balanced = TRUE,
    treatment_probs = 0.5)
}

Whatever settings you do to the parallel workers in the first foreach() call, will not be available on the workers in the second call.

Instead, you want do something like:

assignments = foreach::foreach (i = 1:m, .options.future = list(seed = TRUE)) %dofuture% {
  ## Initiate the GWS Julia engine
  gswdesign::gswdesign_setup()
  
  gswdesign::sample_gs_walk(
    X = X, phi = 0.1, balanced = TRUE,
    treatment_probs = 0.5)
}

That code would run distributed anywhere in the world without you having to make assumption on the workers, as long as the workers have R, future, gswdesign, and Julia installed.

Now, it looks like gswdesign::gswdesign_setup() does not detect if it has already been called. For example, if you do:

gswdesign::gswdesign_setup()
gswdesign::gswdesign_setup()

in the same R session, it will do the same setup twice. Ideally, gswdesign::gswdesign_setup() would detect that it has already been done once, and just skip it if called multiple times. I would recommend that you reach out to https://github.com/fsavje/gswdesign-R/ and propose this. Alternatively, you could ask for a gswdesign::gswdesign_already_setup() so you do could something like:

assignments = foreach::foreach (i = 1:m, .options.future = list(seed = TRUE)) %dofuture% {
  ## Initiate the GWS Julia engine, if not already done
  if (!gswdesign::gswdesign_already_setup()) gswdesign::gswdesign_setup()
  
  gswdesign::sample_gs_walk(
    X = X, phi = 0.1, balanced = TRUE,
    treatment_probs = 0.5)
}

There's an internal gswdesign:::julia_running() they use themselves to test for this, that you could use, but ideally they should provide this in their API.

[nremenyi]

I really appreciate the detailed response. I did feel what I did was a hack, and as you pointed out it won't work with transient workers. I did try to put "gswdesign::gswdesign_setup()" inside the foreach loop, as you recommended, but its very slow, since it will run for each iteration without knowing if it was executed before, and since the whole code itself is fast then there is no point parallelizing it anymore. I also tried to check for if it was run before, or pass some variables/environment ""gswdesign::gswdesign_setup()" but i could not make it work. I will send your comments to the package developer.

Using "future::plan(future::multicore)" works without running the setup script inside of foreach, but I saw it is not recommended to use that, especially that I am using RStudio.

[HenrikBengtsson]

also tried to check for if it was run before, or pass some variables/environment ""gswdesign::gswdesign_setup()" but i could not make it work.

The Futureverse actually makes it hard to retain information on workers. This is by design, to lower the risk for hacky workarounds that works for a specific setup, but then will fail when we move to other parallel backends. There are ideas on the roadmap to support keeping some information around, e.g. caching of data, but it'll be optional and not mandatory. Such features could potentially cover also cases like this, but we're not there yet.

You could retain information in the gswdesign package environment. Looking at their code, one approach would be to use:

gswdesign:::.gsw_intenv$initiated <- TRUE

That will survive as long as gswdesign remains loaded, which it will. This "hack" emulates one way that gswdesign could handle this internally.

Using future::plan(future::multicore) ...

Yes, I suggest staying away from forked parallelization - there are simply to many potential pot holes and we don't always know where they are and new ones risk popping up at any time.